Jul 22 15:02:26.604565 ip-10-0-128-22 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 15:02:26.604579 ip-10-0-128-22 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 15:02:26.604588 ip-10-0-128-22 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 15:02:26.604914 ip-10-0-128-22 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 15:02:36.656620 ip-10-0-128-22 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 15:02:36.656638 ip-10-0-128-22 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7aa4223101664395b5a637e00de01362 -- Jul 22 15:04:33.090726 ip-10-0-128-22 systemd[1]: Starting Kubernetes Kubelet... Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.501162 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507700 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507711 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507714 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507719 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507722 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507725 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507730 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 15:04:33.580461 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507734 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 15:04:33.575142 ip-10-0-128-22 systemd[1]: Started Kubernetes Kubelet. Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507737 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507740 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507743 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507746 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507749 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507752 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507755 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507758 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507760 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507764 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507766 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507769 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507771 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507774 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507777 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507780 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507782 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507785 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507787 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 15:04:33.581635 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507790 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507793 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507795 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507799 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507803 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507807 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507810 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507813 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507829 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507833 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507836 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507838 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507841 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507844 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507846 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507849 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507853 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507857 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507860 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 15:04:33.582434 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507863 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507866 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507869 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507872 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507875 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507877 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507880 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507882 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507885 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507888 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507891 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507893 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507896 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507898 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507901 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507904 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507906 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507910 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507912 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507915 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 15:04:33.583358 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507918 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507920 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507923 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507926 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507929 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507931 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507933 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507936 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507939 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507941 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507944 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507947 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507949 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507951 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507954 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507956 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507959 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507961 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507964 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507966 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 15:04:33.584189 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507969 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507971 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507974 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507977 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507979 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507982 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507985 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507987 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507990 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507992 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507995 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.507997 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508000 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508728 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508734 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508737 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508740 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508744 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508747 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 15:04:33.585028 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508749 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508752 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508754 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508757 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508760 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508763 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508765 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508768 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508771 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508776 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508780 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508783 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508787 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508790 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508793 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508796 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508798 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508801 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508804 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 15:04:33.585591 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508806 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508809 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508811 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508814 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508816 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508819 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508821 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508824 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508826 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508829 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508831 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508834 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508837 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508839 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508841 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508844 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508847 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508850 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508853 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508856 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 15:04:33.586157 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508859 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508861 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508864 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508866 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508869 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508871 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508874 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508876 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508879 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508881 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508884 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508886 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508888 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508891 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508893 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508896 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508898 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508901 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508903 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508906 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 15:04:33.586792 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508908 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508911 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508913 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508915 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508925 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508931 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508935 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508938 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508941 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508943 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508946 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508949 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508951 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508954 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508957 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508960 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508962 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508964 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508967 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508969 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 15:04:33.587538 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508972 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508974 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508977 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508979 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508982 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508984 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508987 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508989 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508992 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508994 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.508997 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509000 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509004 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509007 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509434 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509454 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509464 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509468 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509471 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 15:04:33.588154 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509474 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509476 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509479 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509482 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509485 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509487 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509490 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509492 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509495 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509498 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509500 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509503 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509506 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509508 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509511 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509515 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509520 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509523 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509526 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 15:04:33.588974 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509530 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509533 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509536 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509539 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509542 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509546 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509549 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509552 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509555 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509558 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509560 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509571 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509574 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509577 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509579 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509582 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509585 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509588 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509590 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509593 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 15:04:33.589618 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509595 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509598 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509600 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509603 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509606 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509608 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509611 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509614 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509616 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509619 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509621 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509624 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509626 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509629 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509631 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509634 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509636 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509639 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509641 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509644 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 15:04:33.590307 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509646 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509649 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509651 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509655 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509663 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509666 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509669 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509671 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509676 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509678 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509681 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509684 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509686 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509689 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509691 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509694 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509697 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509700 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509702 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509705 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 15:04:33.591140 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509707 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509710 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509712 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509715 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509717 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509720 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509722 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509725 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509727 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509730 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509732 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509735 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509737 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509740 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.509742 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509823 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509830 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509851 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509856 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509860 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 15:04:33.592036 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509864 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509868 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509872 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509875 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509878 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509882 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509885 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509888 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509891 2580 flags.go:64] FLAG: --cgroup-root="" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509894 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509897 2580 flags.go:64] FLAG: --client-ca-file="" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509899 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509902 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509908 2580 flags.go:64] FLAG: --cluster-domain="" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509911 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509915 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509918 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509921 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509924 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509927 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509930 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509933 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509936 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509939 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 15:04:33.592700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509942 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509945 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509948 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509953 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509956 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509959 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509967 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509970 2580 flags.go:64] FLAG: --enable-server="true" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509975 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509982 2580 flags.go:64] FLAG: --event-burst="100" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509985 2580 flags.go:64] FLAG: --event-qps="50" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509988 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509991 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509994 2580 flags.go:64] FLAG: --eviction-hard="" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.509998 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510000 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510003 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510006 2580 flags.go:64] FLAG: --eviction-soft="" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510009 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510012 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510015 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510018 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510020 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510023 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510026 2580 flags.go:64] FLAG: --feature-gates="" Jul 22 15:04:33.593539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510030 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510033 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510036 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510039 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510042 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510044 2580 flags.go:64] FLAG: --help="false" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510047 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-22.ec2.internal" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510050 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510053 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510056 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510060 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510064 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510066 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510069 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510080 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510085 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510088 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510091 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510094 2580 flags.go:64] FLAG: --kube-reserved="" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510097 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510099 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510102 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510105 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510108 2580 flags.go:64] FLAG: --lock-file="" Jul 22 15:04:33.594349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510111 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510113 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510116 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510121 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510124 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510127 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510130 2580 flags.go:64] FLAG: --logging-format="text" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510132 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510136 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510138 2580 flags.go:64] FLAG: --manifest-url="" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510141 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510145 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510148 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510152 2580 flags.go:64] FLAG: --max-pods="110" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510155 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510157 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510160 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510163 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510167 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510170 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510173 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510179 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510182 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510192 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 15:04:33.595359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510196 2580 flags.go:64] FLAG: --pod-cidr="" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510199 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510203 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510209 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510212 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510215 2580 flags.go:64] FLAG: --port="10250" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510218 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510221 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f6bae574777a450c" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510224 2580 flags.go:64] FLAG: --qos-reserved="" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510227 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510230 2580 flags.go:64] FLAG: --register-node="true" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510233 2580 flags.go:64] FLAG: --register-with-taints="" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510237 2580 flags.go:64] FLAG: --registry-burst="10" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510240 2580 flags.go:64] FLAG: --registry-qps="5" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510243 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510245 2580 flags.go:64] FLAG: --reserved-memory="" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510249 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510252 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510255 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510257 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510260 2580 flags.go:64] FLAG: --runonce="false" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510263 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510266 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510269 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510272 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 15:04:33.741337 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510274 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510277 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510281 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510284 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510287 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510290 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510294 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510302 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510306 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510308 2580 flags.go:64] FLAG: --system-cgroups="" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510311 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510316 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510319 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510322 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510329 2580 flags.go:64] FLAG: --tls-min-version="" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510332 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510335 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510338 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510341 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510344 2580 flags.go:64] FLAG: --v="2" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510347 2580 flags.go:64] FLAG: --version="false" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510351 2580 flags.go:64] FLAG: --vmodule="" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510354 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.510357 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512040 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 15:04:33.759287 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512051 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512055 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512058 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512061 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512064 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512068 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512071 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512074 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512084 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512088 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512092 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512095 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512097 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512102 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512105 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512108 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512110 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512113 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512115 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512118 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 15:04:33.765407 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512121 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512123 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512126 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512129 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512132 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512135 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512137 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512140 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512143 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512146 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512152 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512156 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512161 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512164 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512167 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512169 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512172 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512175 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512178 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 15:04:33.766255 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512181 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512184 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512188 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512191 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512194 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512196 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512201 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512203 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512206 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512208 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512211 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512214 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512217 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512219 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512222 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512225 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512227 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512230 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512232 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512235 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 15:04:33.766893 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512237 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512240 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512243 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512245 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512248 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512250 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512253 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512256 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512258 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512261 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512263 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512266 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512269 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512272 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512275 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512279 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512282 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512285 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512289 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512292 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 15:04:33.767619 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512295 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512298 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512300 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512304 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512306 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512309 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512311 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512314 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512316 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512319 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512322 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512324 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512326 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512329 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512332 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512335 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512337 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512341 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 15:04:33.768213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.512344 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.512353 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.519523 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.519536 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.520484 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.520492 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520553 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520558 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520562 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520565 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520569 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520572 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520575 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520578 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 15:04:33.768815 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520581 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520586 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520589 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520593 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520596 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520599 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520601 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520604 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520607 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520609 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520612 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520614 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520617 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520620 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520622 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520625 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520628 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520630 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520633 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520635 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 15:04:33.769213 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520638 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520641 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520643 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520645 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520648 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520650 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520653 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520655 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520658 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520660 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520663 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520666 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520668 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520673 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520675 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520678 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520682 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520685 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520688 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 15:04:33.769826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520690 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520693 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520696 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520698 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520701 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520704 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520706 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520708 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520711 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520713 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520716 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520718 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520721 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520725 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520729 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520732 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520734 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520738 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520740 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 15:04:33.770421 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520743 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520746 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520748 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520751 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520754 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520756 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520759 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520763 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520766 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520769 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520771 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520776 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520779 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520782 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520785 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520787 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520790 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520792 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520795 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520797 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 15:04:33.773023 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520800 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520802 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520805 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520807 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520810 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520812 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520815 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520818 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520821 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520825 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520829 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520832 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520835 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.520840 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 15:04:33.773627 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520973 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520978 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520981 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520985 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520987 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520992 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520995 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.520998 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521001 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521003 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521006 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521009 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521012 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521014 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521017 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521019 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521022 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521024 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521027 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521029 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 15:04:33.774249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521032 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521035 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521037 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521040 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521042 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521045 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521047 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521049 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521052 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521057 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521060 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521062 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521065 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521067 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521070 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521072 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521075 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521079 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521081 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 15:04:33.774877 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521084 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521087 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521091 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521094 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521097 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521100 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521103 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521106 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521109 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521111 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521113 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521116 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521118 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521121 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521123 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521126 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521128 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521131 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521134 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 15:04:33.775706 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521136 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521138 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521141 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521145 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521148 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521150 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521153 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521155 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521158 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521160 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521162 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521166 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521169 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521172 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521174 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521176 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521179 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521181 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521184 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521187 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 15:04:33.776319 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521190 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521192 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521195 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521197 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521200 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521203 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521205 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521208 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521210 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521213 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521215 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521218 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521221 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521223 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521225 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521231 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521234 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521236 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521239 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521241 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 15:04:33.776937 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:33.521243 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 15:04:33.777552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.521248 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 15:04:33.777552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.521339 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 15:04:33.777552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.523327 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 15:04:33.777552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.524825 2580 server.go:1019] "Starting client certificate rotation" Jul 22 15:04:33.777552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.524924 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 15:04:33.777552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.524954 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 15:04:33.777552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.527170 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 15:04:33.777552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.531778 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 15:04:33.777552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.542760 2580 log.go:25] "Validated CRI v1 runtime API" Jul 22 15:04:33.777552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.544816 2580 log.go:25] "Validated CRI v1 image API" Jul 22 15:04:33.777552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.546623 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 15:04:33.777552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.549276 2580 fs.go:135] Filesystem UUIDs: map[0a2bcd0f-6d8d-4da3-b16b-43795af649c6:/dev/nvme0n1p4 35294590-1013-4b61-be93-32b52e39c8aa:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 22 15:04:33.778017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.549300 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 15:04:33.778091 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.556017 2580 manager.go:216] Machine: {Timestamp:2026-07-22 15:04:33.554908491 +0000 UTC m=+0.338482446 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099837 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22e9c0e3cc774e69471537f939de13 SystemUUID:ec22e9c0-e3cc-774e-6947-1537f939de13 BootID:7aa42231-0166-4395-b5a6-37e00de01362 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3c:3a:be:3e:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3c:3a:be:3e:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:8d:0b:6a:ae:0f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 15:04:33.778091 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.556123 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 15:04:33.778091 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.556197 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 15:04:33.778091 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.556222 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 15:04:33.778091 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.559308 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 15:04:33.778091 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.559331 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-22.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 15:04:33.778091 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.559975 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.559984 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.559991 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.560748 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.561818 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.564561 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.564674 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.564699 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.564708 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.564722 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.564731 2580 kubelet.go:398] "Adding apiserver pod source" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.564738 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.565043 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.565063 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.567973 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.569679 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.569706 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.570692 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.570705 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.570711 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.570716 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.570725 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.570730 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.570744 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.570750 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.570756 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.570763 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.570780 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.570786 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.571468 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.571476 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.574380 2580 server.go:1268] "Started kubelet" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.574436 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.574470 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.574569 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.582704 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.582712 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-22.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.582776 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-22.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.582824 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 15:04:33.778380 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.582889 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8rwh2" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.583620 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.584530 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.587202 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8rwh2" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.589031 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-22.ec2.internal.18c4a4e9236ec74f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-22.ec2.internal,UID:ip-10-0-128-22.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-22.ec2.internal,},FirstTimestamp:2026-07-22 15:04:33.574356815 +0000 UTC m=+0.357930769,LastTimestamp:2026-07-22 15:04:33.574356815 +0000 UTC m=+0.357930769,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-22.ec2.internal,}" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.590465 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.590478 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.590683 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.591054 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.591097 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.591120 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.591223 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.591233 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.591421 2580 factory.go:55] Registering systemd factory Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.591432 2580 factory.go:223] Registration of the systemd container factory successfully Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.591638 2580 factory.go:153] Registering CRI-O factory Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.591661 2580 factory.go:223] Registration of the crio container factory successfully Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.596565 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.596594 2580 factory.go:103] Registering Raw factory Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.596609 2580 manager.go:1196] Started watching for new ooms in manager Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.596968 2580 manager.go:349] Starting recovery of all containers Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.602193 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.604910 2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-22.ec2.internal\" not found" node="ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.606821 2580 manager.go:354] Recovery completed Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.611480 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.613352 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.613379 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.613388 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.614918 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.614941 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.614961 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.615849 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.616309 2580 policy_none.go:49] "None policy: Start" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.616324 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.616335 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.616684 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 22 15:04:35.224066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.616704 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.616718 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.622375 2580 policy_none.go:47] "Start" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.668949 2580 manager.go:337] "Starting Device Plugin manager" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.669098 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.669111 2580 server.go:87] "Starting device plugin registration server" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.669290 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.669299 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.669402 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.669490 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.669500 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.669871 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.669918 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.761518 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.762744 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.762776 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.762798 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.762833 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.765483 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.769624 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.770398 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.770425 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.770436 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.770474 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.778744 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.778767 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-22.ec2.internal\": node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.796568 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.863933 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal"] Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.863988 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.865495 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.865519 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.865528 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.866750 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.866871 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.866897 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.867466 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.867493 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 15:04:35.225489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.867507 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.867510 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.867528 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.867542 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.868673 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.868697 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.869306 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientMemory" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.869331 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.869340 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeHasSufficientPID" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.895823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e61fc359e18a19e5497b5e6df8484f9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-22.ec2.internal\" (UID: \"5e61fc359e18a19e5497b5e6df8484f9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.895843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8cce66671021be49acd4aba890d28bc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal\" (UID: \"d8cce66671021be49acd4aba890d28bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:33.895860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8cce66671021be49acd4aba890d28bc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal\" (UID: \"d8cce66671021be49acd4aba890d28bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.896854 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.909103 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-22.ec2.internal\" not found" node="ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.227318 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:33.912768 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-22.ec2.internal\" not found" node="ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.391055 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.280974 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e61fc359e18a19e5497b5e6df8484f9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-22.ec2.internal\" (UID: \"5e61fc359e18a19e5497b5e6df8484f9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.391055 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.280996 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8cce66671021be49acd4aba890d28bc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal\" (UID: \"d8cce66671021be49acd4aba890d28bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.391055 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.281012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8cce66671021be49acd4aba890d28bc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal\" (UID: \"d8cce66671021be49acd4aba890d28bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.391055 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.281063 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8cce66671021be49acd4aba890d28bc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal\" (UID: \"d8cce66671021be49acd4aba890d28bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.391055 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.281077 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e61fc359e18a19e5497b5e6df8484f9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-22.ec2.internal\" (UID: \"5e61fc359e18a19e5497b5e6df8484f9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.391055 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.281084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8cce66671021be49acd4aba890d28bc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal\" (UID: \"d8cce66671021be49acd4aba890d28bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.391055 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:35.282026 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:35.391055 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:35.382606 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:35.552501 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:35.483052 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:35.552501 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.497207 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.552501 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.501093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" Jul 22 15:04:35.713472 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:35.583409 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:35.713472 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:35.683800 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:35.876270 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:35.783850 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:35.876270 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.810009 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 15:04:35.876270 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.810178 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 15:04:35.876270 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.810189 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 15:04:35.975559 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.881808 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 14:59:34 +0000 UTC" deadline="2028-01-23 11:46:29.253401447 +0000 UTC" Jul 22 15:04:35.975559 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:35.881841 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13196h41m53.371563545s" Jul 22 15:04:35.975559 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:35.884552 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:36.062862 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:35.985135 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:36.181700 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:36.071640 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e61fc359e18a19e5497b5e6df8484f9.slice/crio-943dc2964d32b656bfb7291fb5453e22446e1a964a119461e3e645c363fcebdb WatchSource:0}: Error finding container 943dc2964d32b656bfb7291fb5453e22446e1a964a119461e3e645c363fcebdb: Status 404 returned error can't find the container with id 943dc2964d32b656bfb7291fb5453e22446e1a964a119461e3e645c363fcebdb Jul 22 15:04:36.181700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.075402 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 15:04:36.181700 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:36.086224 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:36.284386 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:36.186611 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:36.284386 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.256305 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 15:04:36.287909 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:36.287184 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-22.ec2.internal\" not found" Jul 22 15:04:36.413652 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.307186 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 15:04:36.413652 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.375979 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" Jul 22 15:04:36.413652 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.387708 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 15:04:36.413652 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.388820 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" Jul 22 15:04:36.413652 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.403748 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.850883 2580 apiserver.go:52] "Watching apiserver" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.852222 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.859606 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.859933 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-bvp4f","kube-system/global-pull-secret-syncer-bjpkw","kube-system/konnectivity-agent-b695k","kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd","openshift-cluster-node-tuning-operator/tuned-bjhfj","openshift-dns/node-resolver-t76c6","openshift-image-registry/node-ca-rrrth","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal","openshift-multus/multus-additional-cni-plugins-5trxb","openshift-multus/multus-t6vrn","openshift-multus/network-metrics-daemon-bwrpd","openshift-network-diagnostics/network-check-target-58ptc","openshift-network-operator/iptables-alerter-wxjcx"] Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.862017 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.863561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:36.863628 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.864650 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b695k" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.866452 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wxjcx" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.867654 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.868531 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.870034 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rrrth" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.870513 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t76c6" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.871690 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:36.871728 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.872918 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.873517 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.873539 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.873647 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.873832 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.873864 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zlw6r\"" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.873886 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.874173 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.874194 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7mbh2\"" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.874226 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 15:04:36.917512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.876291 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.876552 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.876615 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sj7ck\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:36.876678 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.877629 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.877659 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.877862 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.878322 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.878355 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.878868 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.878908 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.879042 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.879054 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.879089 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.879639 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w5sfb\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.879868 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.879888 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.879944 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.880094 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.880154 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.880281 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.884974 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qgcqv\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.885239 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.885295 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.885320 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7xz7d\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.885165 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.885457 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.885719 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ltqnk\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.885742 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5mk7g\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.885753 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 15:04:36.919390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.886101 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4ptp6\"" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.890715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/719c1099-aeb4-4738-ba6f-ae117d6746b6-os-release\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.890746 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.890777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.890795 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ecbe0445-2b23-4b1b-b230-eb80fbb1915e-iptables-alerter-script\") pod \"iptables-alerter-wxjcx\" (UID: \"ecbe0445-2b23-4b1b-b230-eb80fbb1915e\") " pod="openshift-network-operator/iptables-alerter-wxjcx" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.890842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-run-ovn\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.890860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-env-overrides\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.890877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7003d3d9-bb04-433c-a3f4-a5e95f7f93da-hosts-file\") pod \"node-resolver-t76c6\" (UID: \"7003d3d9-bb04-433c-a3f4-a5e95f7f93da\") " pod="openshift-dns/node-resolver-t76c6" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.890929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-run-k8s-cni-cncf-io\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.890956 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-etc-kubernetes\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.890982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4bj\" (UniqueName: \"kubernetes.io/projected/544b0b33-d190-4618-b038-1473f01a2bc4-kube-api-access-2h4bj\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/719c1099-aeb4-4738-ba6f-ae117d6746b6-system-cni-dir\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/719c1099-aeb4-4738-ba6f-ae117d6746b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.920823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-kubelet-config\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891085 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-systemd-units\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891110 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-ovn-node-metrics-cert\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891156 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-ovnkube-script-lib\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-device-dir\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-sys\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e661d076-cf63-4136-9ac5-3afa121bda96-serviceca\") pod \"node-ca-rrrth\" (UID: \"e661d076-cf63-4136-9ac5-3afa121bda96\") " pod="openshift-image-registry/node-ca-rrrth" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891258 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-hostroot\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/719c1099-aeb4-4738-ba6f-ae117d6746b6-cnibin\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-node-log\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891325 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbb7d\" (UniqueName: \"kubernetes.io/projected/ccdb4fa1-83d8-4949-a402-394895fad8c6-kube-api-access-lbb7d\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-host\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/544b0b33-d190-4618-b038-1473f01a2bc4-cni-binary-copy\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-var-lib-cni-bin\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.922246 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891452 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-var-lib-kubelet\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-kubelet\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-etc-openvswitch\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891547 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhpqq\" (UniqueName: \"kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq\") pod \"network-check-target-58ptc\" (UID: \"a9c3f4f1-b506-4b50-b350-a07d4a34cded\") " pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-modprobe-d\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891591 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-dbus\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24cb6d68-e498-4801-b5b9-76ae50893cc4-konnectivity-ca\") pod \"konnectivity-agent-b695k\" (UID: \"24cb6d68-e498-4801-b5b9-76ae50893cc4\") " pod="kube-system/konnectivity-agent-b695k" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-systemd\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891657 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-lib-modules\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/719c1099-aeb4-4738-ba6f-ae117d6746b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-slash\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-run-ovn-kubernetes\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891784 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-ovnkube-config\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.922769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5zcx\" (UniqueName: \"kubernetes.io/projected/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-kube-api-access-v5zcx\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-tuned\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-multus-cni-dir\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-os-release\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk8bt\" (UniqueName: \"kubernetes.io/projected/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-kube-api-access-gk8bt\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pg7h\" (UniqueName: \"kubernetes.io/projected/719c1099-aeb4-4738-ba6f-ae117d6746b6-kube-api-access-7pg7h\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwwnt\" (UniqueName: \"kubernetes.io/projected/ecbe0445-2b23-4b1b-b230-eb80fbb1915e-kube-api-access-kwwnt\") pod \"iptables-alerter-wxjcx\" (UID: \"ecbe0445-2b23-4b1b-b230-eb80fbb1915e\") " pod="openshift-network-operator/iptables-alerter-wxjcx" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-cni-netd\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891951 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-sys-fs\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.891986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-kubernetes\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-run\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.923508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef6ad219-4138-4187-b9d5-c343cb471f44-tmp\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-system-cni-dir\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/719c1099-aeb4-4738-ba6f-ae117d6746b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892217 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24cb6d68-e498-4801-b5b9-76ae50893cc4-agent-certs\") pod \"konnectivity-agent-b695k\" (UID: \"24cb6d68-e498-4801-b5b9-76ae50893cc4\") " pod="kube-system/konnectivity-agent-b695k" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-run-netns\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-socket-dir\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-sysctl-d\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892292 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-sysctl-conf\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892320 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-var-lib-kubelet\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892333 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-log-socket\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fpkn\" (UniqueName: \"kubernetes.io/projected/7003d3d9-bb04-433c-a3f4-a5e95f7f93da-kube-api-access-6fpkn\") pod \"node-resolver-t76c6\" (UID: \"7003d3d9-bb04-433c-a3f4-a5e95f7f93da\") " pod="openshift-dns/node-resolver-t76c6" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892363 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-run-netns\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-var-lib-cni-multus\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892390 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-multus-conf-dir\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.924017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/544b0b33-d190-4618-b038-1473f01a2bc4-multus-daemon-config\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-run-multus-certs\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ecbe0445-2b23-4b1b-b230-eb80fbb1915e-host-slash\") pod \"iptables-alerter-wxjcx\" (UID: \"ecbe0445-2b23-4b1b-b230-eb80fbb1915e\") " pod="openshift-network-operator/iptables-alerter-wxjcx" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-var-lib-openvswitch\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-cni-bin\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-sysconfig\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892524 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxn8p\" (UniqueName: \"kubernetes.io/projected/ef6ad219-4138-4187-b9d5-c343cb471f44-kube-api-access-dxn8p\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892547 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l2xs\" (UniqueName: \"kubernetes.io/projected/e661d076-cf63-4136-9ac5-3afa121bda96-kube-api-access-4l2xs\") pod \"node-ca-rrrth\" (UID: \"e661d076-cf63-4136-9ac5-3afa121bda96\") " pod="openshift-image-registry/node-ca-rrrth" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7003d3d9-bb04-433c-a3f4-a5e95f7f93da-tmp-dir\") pod \"node-resolver-t76c6\" (UID: \"7003d3d9-bb04-433c-a3f4-a5e95f7f93da\") " pod="openshift-dns/node-resolver-t76c6" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892607 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-multus-socket-dir-parent\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892638 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/719c1099-aeb4-4738-ba6f-ae117d6746b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892660 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-run-systemd\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-run-openvswitch\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-registration-dir\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.924607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e661d076-cf63-4136-9ac5-3afa121bda96-host\") pod \"node-ca-rrrth\" (UID: \"e661d076-cf63-4136-9ac5-3afa121bda96\") " pod="openshift-image-registry/node-ca-rrrth" Jul 22 15:04:36.925204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.892736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-cnibin\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.925204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.895415 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 15:04:36.934693 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.934677 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j9gd2" Jul 22 15:04:36.937967 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.937948 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j9gd2" Jul 22 15:04:36.977033 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.977012 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 15:04:36.993234 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-lib-modules\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.993351 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/719c1099-aeb4-4738-ba6f-ae117d6746b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.993351 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-slash\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.993351 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993271 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-run-ovn-kubernetes\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.993351 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-ovnkube-config\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.993351 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5zcx\" (UniqueName: \"kubernetes.io/projected/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-kube-api-access-v5zcx\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.993351 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-tuned\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.993351 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-run-ovn-kubernetes\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.993625 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-slash\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.993625 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993392 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-lib-modules\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.993625 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993544 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-multus-cni-dir\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.993625 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993349 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-multus-cni-dir\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.993625 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-os-release\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.993625 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8bt\" (UniqueName: \"kubernetes.io/projected/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-kube-api-access-gk8bt\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:36.993892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pg7h\" (UniqueName: \"kubernetes.io/projected/719c1099-aeb4-4738-ba6f-ae117d6746b6-kube-api-access-7pg7h\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.993892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwwnt\" (UniqueName: \"kubernetes.io/projected/ecbe0445-2b23-4b1b-b230-eb80fbb1915e-kube-api-access-kwwnt\") pod \"iptables-alerter-wxjcx\" (UID: \"ecbe0445-2b23-4b1b-b230-eb80fbb1915e\") " pod="openshift-network-operator/iptables-alerter-wxjcx" Jul 22 15:04:36.993892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-cni-netd\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.993892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-sys-fs\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.993892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993699 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-os-release\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.993892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:36.993892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.993892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.993892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-kubernetes\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993940 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/719c1099-aeb4-4738-ba6f-ae117d6746b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:36.993964 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994031 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-sys-fs\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:36.994037 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret podName:137d785b-18a1-41d5-9e16-d4c7f5f7af6b nodeName:}" failed. No retries permitted until 2026-07-22 15:04:37.494015399 +0000 UTC m=+2.992915192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret") pod "global-pull-secret-syncer-bjpkw" (UID: "137d785b-18a1-41d5-9e16-d4c7f5f7af6b") : object "kube-system"/"original-pull-secret" not registered Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.993622 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994110 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994229 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994262 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-kubernetes\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-run\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef6ad219-4138-4187-b9d5-c343cb471f44-tmp\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-system-cni-dir\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/719c1099-aeb4-4738-ba6f-ae117d6746b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.994388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994390 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-run\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994433 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-cni-netd\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994443 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-system-cni-dir\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24cb6d68-e498-4801-b5b9-76ae50893cc4-agent-certs\") pod \"konnectivity-agent-b695k\" (UID: \"24cb6d68-e498-4801-b5b9-76ae50893cc4\") " pod="kube-system/konnectivity-agent-b695k" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-run-netns\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-socket-dir\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-sysctl-d\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-sysctl-conf\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994580 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-run-netns\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-var-lib-kubelet\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-log-socket\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-socket-dir\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fpkn\" (UniqueName: \"kubernetes.io/projected/7003d3d9-bb04-433c-a3f4-a5e95f7f93da-kube-api-access-6fpkn\") pod \"node-resolver-t76c6\" (UID: \"7003d3d9-bb04-433c-a3f4-a5e95f7f93da\") " pod="openshift-dns/node-resolver-t76c6" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994687 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-sysctl-d\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-run-netns\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.994949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994726 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-var-lib-kubelet\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994736 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-var-lib-cni-multus\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994754 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-sysctl-conf\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-multus-conf-dir\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-run-netns\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/544b0b33-d190-4618-b038-1473f01a2bc4-multus-daemon-config\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994828 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-log-socket\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994819 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/719c1099-aeb4-4738-ba6f-ae117d6746b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-run-multus-certs\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-run-multus-certs\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994894 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ecbe0445-2b23-4b1b-b230-eb80fbb1915e-host-slash\") pod \"iptables-alerter-wxjcx\" (UID: \"ecbe0445-2b23-4b1b-b230-eb80fbb1915e\") " pod="openshift-network-operator/iptables-alerter-wxjcx" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-var-lib-cni-multus\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994866 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-multus-conf-dir\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ecbe0445-2b23-4b1b-b230-eb80fbb1915e-host-slash\") pod \"iptables-alerter-wxjcx\" (UID: \"ecbe0445-2b23-4b1b-b230-eb80fbb1915e\") " pod="openshift-network-operator/iptables-alerter-wxjcx" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994949 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-ovnkube-config\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.995650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.994999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-var-lib-openvswitch\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-cni-bin\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-sysconfig\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995071 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-var-lib-openvswitch\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995080 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-cni-bin\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxn8p\" (UniqueName: \"kubernetes.io/projected/ef6ad219-4138-4187-b9d5-c343cb471f44-kube-api-access-dxn8p\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995133 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l2xs\" (UniqueName: \"kubernetes.io/projected/e661d076-cf63-4136-9ac5-3afa121bda96-kube-api-access-4l2xs\") pod \"node-ca-rrrth\" (UID: \"e661d076-cf63-4136-9ac5-3afa121bda96\") " pod="openshift-image-registry/node-ca-rrrth" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-sysconfig\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7003d3d9-bb04-433c-a3f4-a5e95f7f93da-tmp-dir\") pod \"node-resolver-t76c6\" (UID: \"7003d3d9-bb04-433c-a3f4-a5e95f7f93da\") " pod="openshift-dns/node-resolver-t76c6" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-multus-socket-dir-parent\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/719c1099-aeb4-4738-ba6f-ae117d6746b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-run-systemd\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995245 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-run-openvswitch\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-registration-dir\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e661d076-cf63-4136-9ac5-3afa121bda96-host\") pod \"node-ca-rrrth\" (UID: \"e661d076-cf63-4136-9ac5-3afa121bda96\") " pod="openshift-image-registry/node-ca-rrrth" Jul 22 15:04:36.996264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-cnibin\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-run-openvswitch\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/719c1099-aeb4-4738-ba6f-ae117d6746b6-os-release\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995394 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/544b0b33-d190-4618-b038-1473f01a2bc4-multus-daemon-config\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995411 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/719c1099-aeb4-4738-ba6f-ae117d6746b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995430 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/719c1099-aeb4-4738-ba6f-ae117d6746b6-os-release\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995456 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-run-systemd\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995459 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ecbe0445-2b23-4b1b-b230-eb80fbb1915e-iptables-alerter-script\") pod \"iptables-alerter-wxjcx\" (UID: \"ecbe0445-2b23-4b1b-b230-eb80fbb1915e\") " pod="openshift-network-operator/iptables-alerter-wxjcx" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:36.995499 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e661d076-cf63-4136-9ac5-3afa121bda96-host\") pod \"node-ca-rrrth\" (UID: \"e661d076-cf63-4136-9ac5-3afa121bda96\") " pod="openshift-image-registry/node-ca-rrrth" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995532 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-cnibin\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995562 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-multus-socket-dir-parent\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.996868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995576 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-registration-dir\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.997476 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995791 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7003d3d9-bb04-433c-a3f4-a5e95f7f93da-tmp-dir\") pod \"node-resolver-t76c6\" (UID: \"7003d3d9-bb04-433c-a3f4-a5e95f7f93da\") " pod="openshift-dns/node-resolver-t76c6" Jul 22 15:04:36.997476 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:36.995808 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs podName:5d3d0c39-a649-4919-aeb9-6566ca9c64f1 nodeName:}" failed. No retries permitted until 2026-07-22 15:04:37.495592474 +0000 UTC m=+2.994492244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs") pod "network-metrics-daemon-bwrpd" (UID: "5d3d0c39-a649-4919-aeb9-6566ca9c64f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:04:36.997476 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.995982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ecbe0445-2b23-4b1b-b230-eb80fbb1915e-iptables-alerter-script\") pod \"iptables-alerter-wxjcx\" (UID: \"ecbe0445-2b23-4b1b-b230-eb80fbb1915e\") " pod="openshift-network-operator/iptables-alerter-wxjcx" Jul 22 15:04:36.997476 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.996068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-run-ovn\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.997476 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.996450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-run-ovn\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.997476 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.996899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-env-overrides\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.997476 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.997149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-env-overrides\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.997800 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.997531 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef6ad219-4138-4187-b9d5-c343cb471f44-tmp\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.997800 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.997536 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7003d3d9-bb04-433c-a3f4-a5e95f7f93da-hosts-file\") pod \"node-resolver-t76c6\" (UID: \"7003d3d9-bb04-433c-a3f4-a5e95f7f93da\") " pod="openshift-dns/node-resolver-t76c6" Jul 22 15:04:36.997800 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.997539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7003d3d9-bb04-433c-a3f4-a5e95f7f93da-hosts-file\") pod \"node-resolver-t76c6\" (UID: \"7003d3d9-bb04-433c-a3f4-a5e95f7f93da\") " pod="openshift-dns/node-resolver-t76c6" Jul 22 15:04:36.997908 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.997807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-run-k8s-cni-cncf-io\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.997908 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.997820 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24cb6d68-e498-4801-b5b9-76ae50893cc4-agent-certs\") pod \"konnectivity-agent-b695k\" (UID: \"24cb6d68-e498-4801-b5b9-76ae50893cc4\") " pod="kube-system/konnectivity-agent-b695k" Jul 22 15:04:36.997908 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.997836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-etc-kubernetes\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.997908 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.997857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-run-k8s-cni-cncf-io\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.997908 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.997859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4bj\" (UniqueName: \"kubernetes.io/projected/544b0b33-d190-4618-b038-1473f01a2bc4-kube-api-access-2h4bj\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.998105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.997905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/719c1099-aeb4-4738-ba6f-ae117d6746b6-system-cni-dir\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.998105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.997955 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-etc-kubernetes\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.998105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.997966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/719c1099-aeb4-4738-ba6f-ae117d6746b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.998105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998001 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-kubelet-config\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:36.998105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-systemd-units\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.998105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/719c1099-aeb4-4738-ba6f-ae117d6746b6-system-cni-dir\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.998105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-ovn-node-metrics-cert\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.998105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998093 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-kubelet-config\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:36.998105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-ovnkube-script-lib\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.998524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-device-dir\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.998524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-sys\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.998524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e661d076-cf63-4136-9ac5-3afa121bda96-serviceca\") pod \"node-ca-rrrth\" (UID: \"e661d076-cf63-4136-9ac5-3afa121bda96\") " pod="openshift-image-registry/node-ca-rrrth" Jul 22 15:04:36.998524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-hostroot\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.998524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/719c1099-aeb4-4738-ba6f-ae117d6746b6-cnibin\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.998524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998291 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ccdb4fa1-83d8-4949-a402-394895fad8c6-device-dir\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.998524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-node-log\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.998524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbb7d\" (UniqueName: \"kubernetes.io/projected/ccdb4fa1-83d8-4949-a402-394895fad8c6-kube-api-access-lbb7d\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:36.998524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-host\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.998524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/544b0b33-d190-4618-b038-1473f01a2bc4-cni-binary-copy\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.998524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-var-lib-cni-bin\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.998524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/719c1099-aeb4-4738-ba6f-ae117d6746b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-var-lib-kubelet\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998570 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-kubelet\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-var-lib-kubelet\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-etc-openvswitch\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpqq\" (UniqueName: \"kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq\") pod \"network-check-target-58ptc\" (UID: \"a9c3f4f1-b506-4b50-b350-a07d4a34cded\") " pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998636 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-sys\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-modprobe-d\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998679 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-dbus\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24cb6d68-e498-4801-b5b9-76ae50893cc4-konnectivity-ca\") pod \"konnectivity-agent-b695k\" (UID: \"24cb6d68-e498-4801-b5b9-76ae50893cc4\") " pod="kube-system/konnectivity-agent-b695k" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-systemd\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-ovnkube-script-lib\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-systemd-units\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-systemd\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998834 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-host-kubelet\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.999066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998876 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-node-log\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.999738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998874 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/719c1099-aeb4-4738-ba6f-ae117d6746b6-cnibin\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:36.999738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998915 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-hostroot\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.999738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-etc-openvswitch\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:36.999738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.998984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-modprobe-d\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.999738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.999025 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e661d076-cf63-4136-9ac5-3afa121bda96-serviceca\") pod \"node-ca-rrrth\" (UID: \"e661d076-cf63-4136-9ac5-3afa121bda96\") " pod="openshift-image-registry/node-ca-rrrth" Jul 22 15:04:36.999738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.999073 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-dbus\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:36.999738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.999227 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef6ad219-4138-4187-b9d5-c343cb471f44-host\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:36.999738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.999412 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/544b0b33-d190-4618-b038-1473f01a2bc4-cni-binary-copy\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:36.999738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.999564 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24cb6d68-e498-4801-b5b9-76ae50893cc4-konnectivity-ca\") pod \"konnectivity-agent-b695k\" (UID: \"24cb6d68-e498-4801-b5b9-76ae50893cc4\") " pod="kube-system/konnectivity-agent-b695k" Jul 22 15:04:36.999738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:36.999113 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/544b0b33-d190-4618-b038-1473f01a2bc4-host-var-lib-cni-bin\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:37.000257 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.000231 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-ovn-node-metrics-cert\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:37.003552 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.003530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef6ad219-4138-4187-b9d5-c343cb471f44-etc-tuned\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:37.005358 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.005261 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fpkn\" (UniqueName: \"kubernetes.io/projected/7003d3d9-bb04-433c-a3f4-a5e95f7f93da-kube-api-access-6fpkn\") pod \"node-resolver-t76c6\" (UID: \"7003d3d9-bb04-433c-a3f4-a5e95f7f93da\") " pod="openshift-dns/node-resolver-t76c6" Jul 22 15:04:37.006909 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.006884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l2xs\" (UniqueName: \"kubernetes.io/projected/e661d076-cf63-4136-9ac5-3afa121bda96-kube-api-access-4l2xs\") pod \"node-ca-rrrth\" (UID: \"e661d076-cf63-4136-9ac5-3afa121bda96\") " pod="openshift-image-registry/node-ca-rrrth" Jul 22 15:04:37.007058 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.007033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pg7h\" (UniqueName: \"kubernetes.io/projected/719c1099-aeb4-4738-ba6f-ae117d6746b6-kube-api-access-7pg7h\") pod \"multus-additional-cni-plugins-5trxb\" (UID: \"719c1099-aeb4-4738-ba6f-ae117d6746b6\") " pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:37.007992 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.007964 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwwnt\" (UniqueName: \"kubernetes.io/projected/ecbe0445-2b23-4b1b-b230-eb80fbb1915e-kube-api-access-kwwnt\") pod \"iptables-alerter-wxjcx\" (UID: \"ecbe0445-2b23-4b1b-b230-eb80fbb1915e\") " pod="openshift-network-operator/iptables-alerter-wxjcx" Jul 22 15:04:37.008554 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.008532 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxn8p\" (UniqueName: \"kubernetes.io/projected/ef6ad219-4138-4187-b9d5-c343cb471f44-kube-api-access-dxn8p\") pod \"tuned-bjhfj\" (UID: \"ef6ad219-4138-4187-b9d5-c343cb471f44\") " pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:37.008921 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.008895 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk8bt\" (UniqueName: \"kubernetes.io/projected/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-kube-api-access-gk8bt\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:37.009105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.009080 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5zcx\" (UniqueName: \"kubernetes.io/projected/1734f3d9-8a01-419e-b81e-0ca3d0fbef10-kube-api-access-v5zcx\") pod \"ovnkube-node-bvp4f\" (UID: \"1734f3d9-8a01-419e-b81e-0ca3d0fbef10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:37.009386 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:37.009371 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 15:04:37.009446 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:37.009389 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 15:04:37.009446 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:37.009398 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhpqq for pod openshift-network-diagnostics/network-check-target-58ptc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:04:37.009505 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:37.009449 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq podName:a9c3f4f1-b506-4b50-b350-a07d4a34cded nodeName:}" failed. No retries permitted until 2026-07-22 15:04:37.509433944 +0000 UTC m=+3.008333721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mhpqq" (UniqueName: "kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq") pod "network-check-target-58ptc" (UID: "a9c3f4f1-b506-4b50-b350-a07d4a34cded") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:04:37.009505 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.009454 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4bj\" (UniqueName: \"kubernetes.io/projected/544b0b33-d190-4618-b038-1473f01a2bc4-kube-api-access-2h4bj\") pod \"multus-t6vrn\" (UID: \"544b0b33-d190-4618-b038-1473f01a2bc4\") " pod="openshift-multus/multus-t6vrn" Jul 22 15:04:37.010921 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.010905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbb7d\" (UniqueName: \"kubernetes.io/projected/ccdb4fa1-83d8-4949-a402-394895fad8c6-kube-api-access-lbb7d\") pod \"aws-ebs-csi-driver-node-g7nsd\" (UID: \"ccdb4fa1-83d8-4949-a402-394895fad8c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:37.052610 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.052558 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" event={"ID":"5e61fc359e18a19e5497b5e6df8484f9","Type":"ContainerStarted","Data":"943dc2964d32b656bfb7291fb5453e22446e1a964a119461e3e645c363fcebdb"} Jul 22 15:04:37.171715 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.171653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5trxb" Jul 22 15:04:37.177345 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.177326 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b695k" Jul 22 15:04:37.180291 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:37.180267 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719c1099_aeb4_4738_ba6f_ae117d6746b6.slice/crio-31eebcf5b766d80e7af07c0d4ad800c9126fc89fde0ede702d518de41f19f0a0 WatchSource:0}: Error finding container 31eebcf5b766d80e7af07c0d4ad800c9126fc89fde0ede702d518de41f19f0a0: Status 404 returned error can't find the container with id 31eebcf5b766d80e7af07c0d4ad800c9126fc89fde0ede702d518de41f19f0a0 Jul 22 15:04:37.185211 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:37.185192 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24cb6d68_e498_4801_b5b9_76ae50893cc4.slice/crio-c068a598c2e6cea0cc5bf7d3a8997061dced0a2e8fd2f8c895fc419a502c4fd3 WatchSource:0}: Error finding container c068a598c2e6cea0cc5bf7d3a8997061dced0a2e8fd2f8c895fc419a502c4fd3: Status 404 returned error can't find the container with id c068a598c2e6cea0cc5bf7d3a8997061dced0a2e8fd2f8c895fc419a502c4fd3 Jul 22 15:04:37.187985 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.187965 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wxjcx" Jul 22 15:04:37.190957 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.190939 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" Jul 22 15:04:37.196312 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:37.196273 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbe0445_2b23_4b1b_b230_eb80fbb1915e.slice/crio-dcaeb108d510cb1e1a380e53f50c00471e47e68948ff0f79ed3758dd0353b695 WatchSource:0}: Error finding container dcaeb108d510cb1e1a380e53f50c00471e47e68948ff0f79ed3758dd0353b695: Status 404 returned error can't find the container with id dcaeb108d510cb1e1a380e53f50c00471e47e68948ff0f79ed3758dd0353b695 Jul 22 15:04:37.197075 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.197058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" Jul 22 15:04:37.198568 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:37.198550 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdb4fa1_83d8_4949_a402_394895fad8c6.slice/crio-f21d979f86cce6849dacb4b3ab030ead2f3027ef865bd174469a7a960eeffac8 WatchSource:0}: Error finding container f21d979f86cce6849dacb4b3ab030ead2f3027ef865bd174469a7a960eeffac8: Status 404 returned error can't find the container with id f21d979f86cce6849dacb4b3ab030ead2f3027ef865bd174469a7a960eeffac8 Jul 22 15:04:37.204720 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:37.204702 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6ad219_4138_4187_b9d5_c343cb471f44.slice/crio-3de671d4891483d0dabed898821013249ebf148ed97ce28a6f5a155fae45b1f3 WatchSource:0}: Error finding container 3de671d4891483d0dabed898821013249ebf148ed97ce28a6f5a155fae45b1f3: Status 404 returned error can't find the container with id 3de671d4891483d0dabed898821013249ebf148ed97ce28a6f5a155fae45b1f3 Jul 22 15:04:37.205892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.205820 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rrrth" Jul 22 15:04:37.211541 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.211523 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t76c6" Jul 22 15:04:37.213026 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:37.213004 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode661d076_cf63_4136_9ac5_3afa121bda96.slice/crio-1f1f81d9542852ab60d64814e539596e1ff39627c1c4300ecd0ba80f6abce3b8 WatchSource:0}: Error finding container 1f1f81d9542852ab60d64814e539596e1ff39627c1c4300ecd0ba80f6abce3b8: Status 404 returned error can't find the container with id 1f1f81d9542852ab60d64814e539596e1ff39627c1c4300ecd0ba80f6abce3b8 Jul 22 15:04:37.217202 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.217186 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:04:37.218530 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:37.218509 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7003d3d9_bb04_433c_a3f4_a5e95f7f93da.slice/crio-99b10dc124cae4fee4b693054dccd412c6a815c785235f82aac4a26f072fd276 WatchSource:0}: Error finding container 99b10dc124cae4fee4b693054dccd412c6a815c785235f82aac4a26f072fd276: Status 404 returned error can't find the container with id 99b10dc124cae4fee4b693054dccd412c6a815c785235f82aac4a26f072fd276 Jul 22 15:04:37.221151 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.221131 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t6vrn" Jul 22 15:04:37.224030 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:37.224013 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1734f3d9_8a01_419e_b81e_0ca3d0fbef10.slice/crio-6f388ff96e1678995023b25975eeaa5f36451fbda29db20585840f0f538fb0a2 WatchSource:0}: Error finding container 6f388ff96e1678995023b25975eeaa5f36451fbda29db20585840f0f538fb0a2: Status 404 returned error can't find the container with id 6f388ff96e1678995023b25975eeaa5f36451fbda29db20585840f0f538fb0a2 Jul 22 15:04:37.228233 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:04:37.228213 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544b0b33_d190_4618_b038_1473f01a2bc4.slice/crio-2226a05789893590c5309882bf3e974774547a7efaa371c3510696462cfa66bb WatchSource:0}: Error finding container 2226a05789893590c5309882bf3e974774547a7efaa371c3510696462cfa66bb: Status 404 returned error can't find the container with id 2226a05789893590c5309882bf3e974774547a7efaa371c3510696462cfa66bb Jul 22 15:04:37.344518 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.344495 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 15:04:37.501654 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.501582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:37.501654 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.501646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:37.501828 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:37.501718 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:04:37.501828 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:37.501745 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 15:04:37.501828 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:37.501773 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs podName:5d3d0c39-a649-4919-aeb9-6566ca9c64f1 nodeName:}" failed. No retries permitted until 2026-07-22 15:04:38.501757927 +0000 UTC m=+4.000657681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs") pod "network-metrics-daemon-bwrpd" (UID: "5d3d0c39-a649-4919-aeb9-6566ca9c64f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:04:37.501828 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:37.501790 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret podName:137d785b-18a1-41d5-9e16-d4c7f5f7af6b nodeName:}" failed. No retries permitted until 2026-07-22 15:04:38.501782185 +0000 UTC m=+4.000681946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret") pod "global-pull-secret-syncer-bjpkw" (UID: "137d785b-18a1-41d5-9e16-d4c7f5f7af6b") : object "kube-system"/"original-pull-secret" not registered Jul 22 15:04:37.602583 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.602548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpqq\" (UniqueName: \"kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq\") pod \"network-check-target-58ptc\" (UID: \"a9c3f4f1-b506-4b50-b350-a07d4a34cded\") " pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:37.602761 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:37.602738 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 15:04:37.602820 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:37.602769 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 15:04:37.602820 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:37.602783 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhpqq for pod openshift-network-diagnostics/network-check-target-58ptc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:04:37.602915 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:37.602840 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq podName:a9c3f4f1-b506-4b50-b350-a07d4a34cded nodeName:}" failed. No retries permitted until 2026-07-22 15:04:38.602818927 +0000 UTC m=+4.101718699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhpqq" (UniqueName: "kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq") pod "network-check-target-58ptc" (UID: "a9c3f4f1-b506-4b50-b350-a07d4a34cded") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:04:37.940001 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.939923 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 14:59:36 +0000 UTC" deadline="2028-01-22 06:53:38.683847325 +0000 UTC" Jul 22 15:04:37.940001 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:37.939954 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13167h49m0.743896576s" Jul 22 15:04:38.060669 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.060529 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t76c6" event={"ID":"7003d3d9-bb04-433c-a3f4-a5e95f7f93da","Type":"ContainerStarted","Data":"99b10dc124cae4fee4b693054dccd412c6a815c785235f82aac4a26f072fd276"} Jul 22 15:04:38.066254 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.066175 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rrrth" event={"ID":"e661d076-cf63-4136-9ac5-3afa121bda96","Type":"ContainerStarted","Data":"1f1f81d9542852ab60d64814e539596e1ff39627c1c4300ecd0ba80f6abce3b8"} Jul 22 15:04:38.068644 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.068525 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" event={"ID":"ccdb4fa1-83d8-4949-a402-394895fad8c6","Type":"ContainerStarted","Data":"f21d979f86cce6849dacb4b3ab030ead2f3027ef865bd174469a7a960eeffac8"} Jul 22 15:04:38.070955 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.070855 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t6vrn" event={"ID":"544b0b33-d190-4618-b038-1473f01a2bc4","Type":"ContainerStarted","Data":"2226a05789893590c5309882bf3e974774547a7efaa371c3510696462cfa66bb"} Jul 22 15:04:38.073213 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.073060 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" event={"ID":"1734f3d9-8a01-419e-b81e-0ca3d0fbef10","Type":"ContainerStarted","Data":"6f388ff96e1678995023b25975eeaa5f36451fbda29db20585840f0f538fb0a2"} Jul 22 15:04:38.076386 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.076359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" event={"ID":"ef6ad219-4138-4187-b9d5-c343cb471f44","Type":"ContainerStarted","Data":"3de671d4891483d0dabed898821013249ebf148ed97ce28a6f5a155fae45b1f3"} Jul 22 15:04:38.079749 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.079680 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wxjcx" event={"ID":"ecbe0445-2b23-4b1b-b230-eb80fbb1915e","Type":"ContainerStarted","Data":"dcaeb108d510cb1e1a380e53f50c00471e47e68948ff0f79ed3758dd0353b695"} Jul 22 15:04:38.088555 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.088338 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b695k" event={"ID":"24cb6d68-e498-4801-b5b9-76ae50893cc4","Type":"ContainerStarted","Data":"c068a598c2e6cea0cc5bf7d3a8997061dced0a2e8fd2f8c895fc419a502c4fd3"} Jul 22 15:04:38.096885 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.096862 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5trxb" event={"ID":"719c1099-aeb4-4738-ba6f-ae117d6746b6","Type":"ContainerStarted","Data":"31eebcf5b766d80e7af07c0d4ad800c9126fc89fde0ede702d518de41f19f0a0"} Jul 22 15:04:38.511606 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.511505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:38.511606 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.511586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:38.511816 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:38.511653 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 15:04:38.511816 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:38.511689 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:04:38.511816 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:38.511720 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret podName:137d785b-18a1-41d5-9e16-d4c7f5f7af6b nodeName:}" failed. No retries permitted until 2026-07-22 15:04:40.511701721 +0000 UTC m=+6.010601493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret") pod "global-pull-secret-syncer-bjpkw" (UID: "137d785b-18a1-41d5-9e16-d4c7f5f7af6b") : object "kube-system"/"original-pull-secret" not registered Jul 22 15:04:38.511816 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:38.511740 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs podName:5d3d0c39-a649-4919-aeb9-6566ca9c64f1 nodeName:}" failed. No retries permitted until 2026-07-22 15:04:40.511730057 +0000 UTC m=+6.010629812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs") pod "network-metrics-daemon-bwrpd" (UID: "5d3d0c39-a649-4919-aeb9-6566ca9c64f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:04:38.612594 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.612555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpqq\" (UniqueName: \"kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq\") pod \"network-check-target-58ptc\" (UID: \"a9c3f4f1-b506-4b50-b350-a07d4a34cded\") " pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:38.612769 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:38.612716 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 15:04:38.612769 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:38.612740 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 15:04:38.612769 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:38.612761 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhpqq for pod openshift-network-diagnostics/network-check-target-58ptc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:04:38.612924 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:38.612815 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq podName:a9c3f4f1-b506-4b50-b350-a07d4a34cded nodeName:}" failed. No retries permitted until 2026-07-22 15:04:40.612797801 +0000 UTC m=+6.111697559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhpqq" (UniqueName: "kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq") pod "network-check-target-58ptc" (UID: "a9c3f4f1-b506-4b50-b350-a07d4a34cded") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:04:38.940598 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.940482 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 14:59:36 +0000 UTC" deadline="2028-02-25 22:42:22.826438978 +0000 UTC" Jul 22 15:04:38.940598 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:38.940545 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13999h37m43.88589906s" Jul 22 15:04:39.051482 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:39.050621 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:39.051482 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:39.050752 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:04:39.051482 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:39.051161 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:39.051482 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:39.051251 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:04:39.051482 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:39.051324 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:39.051482 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:39.051427 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:04:39.125311 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:39.125239 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" event={"ID":"d8cce66671021be49acd4aba890d28bc","Type":"ContainerStarted","Data":"76a765d81dca2e2764cd36d5a7f59f9b7cb2b535d50c3f947cf38eab397b5833"} Jul 22 15:04:40.531074 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:40.530833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:40.531074 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:40.530921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:40.531074 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:40.531000 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:04:40.531074 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:40.531025 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 15:04:40.531074 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:40.531073 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs podName:5d3d0c39-a649-4919-aeb9-6566ca9c64f1 nodeName:}" failed. No retries permitted until 2026-07-22 15:04:44.531051977 +0000 UTC m=+10.029951749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs") pod "network-metrics-daemon-bwrpd" (UID: "5d3d0c39-a649-4919-aeb9-6566ca9c64f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:04:40.531761 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:40.531094 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret podName:137d785b-18a1-41d5-9e16-d4c7f5f7af6b nodeName:}" failed. No retries permitted until 2026-07-22 15:04:44.531083502 +0000 UTC m=+10.029983270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret") pod "global-pull-secret-syncer-bjpkw" (UID: "137d785b-18a1-41d5-9e16-d4c7f5f7af6b") : object "kube-system"/"original-pull-secret" not registered Jul 22 15:04:40.632659 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:40.631998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpqq\" (UniqueName: \"kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq\") pod \"network-check-target-58ptc\" (UID: \"a9c3f4f1-b506-4b50-b350-a07d4a34cded\") " pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:40.632659 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:40.632212 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 15:04:40.632659 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:40.632233 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 15:04:40.632659 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:40.632245 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhpqq for pod openshift-network-diagnostics/network-check-target-58ptc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:04:40.632659 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:40.632297 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq podName:a9c3f4f1-b506-4b50-b350-a07d4a34cded nodeName:}" failed. No retries permitted until 2026-07-22 15:04:44.632279369 +0000 UTC m=+10.131179142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhpqq" (UniqueName: "kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq") pod "network-check-target-58ptc" (UID: "a9c3f4f1-b506-4b50-b350-a07d4a34cded") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:04:41.052268 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:41.052240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:41.052434 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:41.052362 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:04:41.052624 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:41.052530 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:41.052624 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:41.052542 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:41.052624 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:41.052616 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:04:41.052810 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:41.052742 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:04:43.048390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:43.047862 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:43.048390 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:43.048010 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:04:43.048390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:43.048324 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:43.048920 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:43.048412 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:04:43.048920 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:43.048493 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:43.048920 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:43.048588 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:04:44.562666 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:44.562629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:44.563156 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:44.562714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:44.563156 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:44.562816 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 15:04:44.563156 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:44.562817 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:04:44.563156 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:44.562881 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret podName:137d785b-18a1-41d5-9e16-d4c7f5f7af6b nodeName:}" failed. No retries permitted until 2026-07-22 15:04:52.562860143 +0000 UTC m=+18.061759911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret") pod "global-pull-secret-syncer-bjpkw" (UID: "137d785b-18a1-41d5-9e16-d4c7f5f7af6b") : object "kube-system"/"original-pull-secret" not registered Jul 22 15:04:44.563156 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:44.562953 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs podName:5d3d0c39-a649-4919-aeb9-6566ca9c64f1 nodeName:}" failed. No retries permitted until 2026-07-22 15:04:52.56293518 +0000 UTC m=+18.061834942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs") pod "network-metrics-daemon-bwrpd" (UID: "5d3d0c39-a649-4919-aeb9-6566ca9c64f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:04:44.664582 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:44.664015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpqq\" (UniqueName: \"kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq\") pod \"network-check-target-58ptc\" (UID: \"a9c3f4f1-b506-4b50-b350-a07d4a34cded\") " pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:44.664582 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:44.664180 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 15:04:44.664582 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:44.664198 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 15:04:44.664582 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:44.664211 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhpqq for pod openshift-network-diagnostics/network-check-target-58ptc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:04:44.664582 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:44.664263 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq podName:a9c3f4f1-b506-4b50-b350-a07d4a34cded nodeName:}" failed. No retries permitted until 2026-07-22 15:04:52.664246063 +0000 UTC m=+18.163145821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhpqq" (UniqueName: "kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq") pod "network-check-target-58ptc" (UID: "a9c3f4f1-b506-4b50-b350-a07d4a34cded") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:04:45.052207 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:45.051718 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:45.052207 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:45.051829 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:04:45.052207 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:45.051847 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:45.052207 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:45.051924 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:04:45.052207 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:45.051967 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:45.052207 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:45.052038 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:04:47.051815 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:47.051780 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:47.052252 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:47.051791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:47.052252 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:47.051888 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:04:47.052252 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:47.051984 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:04:47.052252 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:47.051798 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:47.052252 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:47.052170 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:04:49.052913 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:49.052733 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:49.052913 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:49.052901 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:04:49.053391 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:49.052994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:49.053391 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:49.053091 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:04:49.053391 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:49.053133 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:49.053391 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:49.053224 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:04:51.047948 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:51.047911 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:51.048408 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:51.047915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:51.048408 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:51.047916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:51.048408 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:51.048047 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:04:51.048408 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:51.048167 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:04:51.048408 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:51.048224 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:04:52.620026 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:52.619988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:52.620507 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:52.620055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:52.620507 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:52.620143 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 15:04:52.620507 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:52.620209 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:04:52.620507 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:52.620220 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret podName:137d785b-18a1-41d5-9e16-d4c7f5f7af6b nodeName:}" failed. No retries permitted until 2026-07-22 15:05:08.620197704 +0000 UTC m=+34.119097493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret") pod "global-pull-secret-syncer-bjpkw" (UID: "137d785b-18a1-41d5-9e16-d4c7f5f7af6b") : object "kube-system"/"original-pull-secret" not registered Jul 22 15:04:52.620507 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:52.620260 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs podName:5d3d0c39-a649-4919-aeb9-6566ca9c64f1 nodeName:}" failed. No retries permitted until 2026-07-22 15:05:08.620244578 +0000 UTC m=+34.119144345 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs") pod "network-metrics-daemon-bwrpd" (UID: "5d3d0c39-a649-4919-aeb9-6566ca9c64f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:04:52.720379 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:52.720347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpqq\" (UniqueName: \"kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq\") pod \"network-check-target-58ptc\" (UID: \"a9c3f4f1-b506-4b50-b350-a07d4a34cded\") " pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:52.720547 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:52.720520 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 15:04:52.720547 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:52.720545 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 15:04:52.720640 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:52.720559 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhpqq for pod openshift-network-diagnostics/network-check-target-58ptc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:04:52.720640 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:52.720616 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq podName:a9c3f4f1-b506-4b50-b350-a07d4a34cded nodeName:}" failed. No retries permitted until 2026-07-22 15:05:08.720601855 +0000 UTC m=+34.219501608 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhpqq" (UniqueName: "kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq") pod "network-check-target-58ptc" (UID: "a9c3f4f1-b506-4b50-b350-a07d4a34cded") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:04:53.047885 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:53.047808 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:53.048028 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:53.047818 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:53.048028 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:53.047933 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:04:53.048028 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:53.047946 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:53.048028 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:53.048022 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:04:53.048252 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:53.048088 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:04:55.048666 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:55.048630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:55.048666 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:55.048669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:55.049154 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:55.048733 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:04:55.049154 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:55.048638 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:55.049154 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:55.048850 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:04:55.049154 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:55.048952 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:04:57.048669 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:57.048630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:57.049241 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:57.048630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:57.049241 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:57.048779 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:04:57.049241 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:57.048645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:57.049241 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:57.048841 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:04:57.049241 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:57.048920 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:04:58.167070 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.166862 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t6vrn" event={"ID":"544b0b33-d190-4618-b038-1473f01a2bc4","Type":"ContainerStarted","Data":"cc1e6fb33194af383d07d9b489fb0c311399317ca99b71257fff2703ac153d68"} Jul 22 15:04:58.171064 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.171028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" event={"ID":"1734f3d9-8a01-419e-b81e-0ca3d0fbef10","Type":"ContainerStarted","Data":"e2815e38cf95af989eedb9aabc4662608ca165fa2a7518ed0479526caf12d70d"} Jul 22 15:04:58.171193 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.171135 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" event={"ID":"1734f3d9-8a01-419e-b81e-0ca3d0fbef10","Type":"ContainerStarted","Data":"160108f399d395b9b51496e86de748fef9115f43823bad5bc6a1e801b089db04"} Jul 22 15:04:58.171193 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.171188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" event={"ID":"1734f3d9-8a01-419e-b81e-0ca3d0fbef10","Type":"ContainerStarted","Data":"6b1955a50550d2f4805bd9b65e0b2342f862dae264d45445c054e83fb720526e"} Jul 22 15:04:58.171281 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.171235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" event={"ID":"1734f3d9-8a01-419e-b81e-0ca3d0fbef10","Type":"ContainerStarted","Data":"7cbf780383cd6c004fe11d122c090d643f7743940475ee4f09c47cab356d81a5"} Jul 22 15:04:58.171327 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.171281 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" event={"ID":"1734f3d9-8a01-419e-b81e-0ca3d0fbef10","Type":"ContainerStarted","Data":"e31997cac2b3448218e25a2f80ce8338b68cfdf2a2a19b944d9aba3a681a65ad"} Jul 22 15:04:58.171360 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.171333 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" event={"ID":"1734f3d9-8a01-419e-b81e-0ca3d0fbef10","Type":"ContainerStarted","Data":"10e62ecf6586281c6776f8fdc6526a11513bed88c62d4c0a46912ad9707fb26c"} Jul 22 15:04:58.173062 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.173004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" event={"ID":"ef6ad219-4138-4187-b9d5-c343cb471f44","Type":"ContainerStarted","Data":"f04145580d4a5a4dc5285a79ef9c7af4fafd5896b71d3ef19be2c300c2812b99"} Jul 22 15:04:58.174765 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.174715 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b695k" event={"ID":"24cb6d68-e498-4801-b5b9-76ae50893cc4","Type":"ContainerStarted","Data":"120b6af930bee8388eb82f82b38189630abec0c0a545ef5488f0d381f790f518"} Jul 22 15:04:58.176466 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.176443 2580 generic.go:356] "Generic (PLEG): container finished" podID="719c1099-aeb4-4738-ba6f-ae117d6746b6" containerID="d762d2654d57366af5a8666c0810b25f09a608e781b6dd8fce16ae455b31f010" exitCode=0 Jul 22 15:04:58.176573 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.176522 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5trxb" event={"ID":"719c1099-aeb4-4738-ba6f-ae117d6746b6","Type":"ContainerDied","Data":"d762d2654d57366af5a8666c0810b25f09a608e781b6dd8fce16ae455b31f010"} Jul 22 15:04:58.180300 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.178491 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" event={"ID":"5e61fc359e18a19e5497b5e6df8484f9","Type":"ContainerStarted","Data":"2fc78cbc7a8eaa520362947388d530f69be21ed300e770d976290b4b901b4399"} Jul 22 15:04:58.183617 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.183577 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t76c6" event={"ID":"7003d3d9-bb04-433c-a3f4-a5e95f7f93da","Type":"ContainerStarted","Data":"406b4107c13e1db671038b3155fef5ae813d1d2b5e42aa549501b91022b36000"} Jul 22 15:04:58.185574 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.185521 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rrrth" event={"ID":"e661d076-cf63-4136-9ac5-3afa121bda96","Type":"ContainerStarted","Data":"007fdc4278d9710255d06117b65795cd3bb4bc4584aa25add42f607016c2baa1"} Jul 22 15:04:58.187302 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.187282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" event={"ID":"ccdb4fa1-83d8-4949-a402-394895fad8c6","Type":"ContainerStarted","Data":"2d1bd3ac0af802995941b3fe86e17d8cb460e2a2444f55d13080b2d6458ed051"} Jul 22 15:04:58.188912 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.188889 2580 generic.go:356] "Generic (PLEG): container finished" podID="d8cce66671021be49acd4aba890d28bc" containerID="724a89745f7f743fbed1a94f2e6f9b10ace6222653a71f187f4a27bd5b508e1f" exitCode=0 Jul 22 15:04:58.189008 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.188923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" event={"ID":"d8cce66671021be49acd4aba890d28bc","Type":"ContainerDied","Data":"724a89745f7f743fbed1a94f2e6f9b10ace6222653a71f187f4a27bd5b508e1f"} Jul 22 15:04:58.191470 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.191403 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t6vrn" podStartSLOduration=3.025920947 podStartE2EDuration="23.191387442s" totalImagesPullingTime="20.165466495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:04:58.1911581 +0000 UTC m=+23.690057874" watchObservedRunningTime="2026-07-22 15:04:58.191387442 +0000 UTC m=+23.690287220" Jul 22 15:04:58.256485 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.255816 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b695k" podStartSLOduration=3.120702715 podStartE2EDuration="23.255796101s" totalImagesPullingTime="20.135093386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:04:58.255607243 +0000 UTC m=+23.754507022" watchObservedRunningTime="2026-07-22 15:04:58.255796101 +0000 UTC m=+23.754695877" Jul 22 15:04:58.280072 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.280018 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rrrth" podStartSLOduration=3.167643668 podStartE2EDuration="23.279998746s" totalImagesPullingTime="20.112355078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:04:58.279707062 +0000 UTC m=+23.778606839" watchObservedRunningTime="2026-07-22 15:04:58.279998746 +0000 UTC m=+23.778898524" Jul 22 15:04:58.302081 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.302040 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t76c6" podStartSLOduration=3.180086283 podStartE2EDuration="23.302026144s" totalImagesPullingTime="20.121939861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:04:58.301612835 +0000 UTC m=+23.800512612" watchObservedRunningTime="2026-07-22 15:04:58.302026144 +0000 UTC m=+23.800925920" Jul 22 15:04:58.366838 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:58.366801 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-22.ec2.internal" podStartSLOduration=22.366786832 podStartE2EDuration="22.366786832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:04:58.335244791 +0000 UTC m=+23.834144558" watchObservedRunningTime="2026-07-22 15:04:58.366786832 +0000 UTC m=+23.865686655" Jul 22 15:04:59.048133 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.048091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:04:59.048347 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:59.048242 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:04:59.048347 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.048318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:04:59.048512 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:59.048387 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:04:59.048512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.048318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:04:59.048512 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:04:59.048477 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:04:59.059217 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.059187 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 15:04:59.193414 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.193334 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" event={"ID":"ccdb4fa1-83d8-4949-a402-394895fad8c6","Type":"ContainerStarted","Data":"d1c3773c11277b10a3ab7edeab6c650a124a7966a4390bae60abf81cafff9021"} Jul 22 15:04:59.195460 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.195417 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" event={"ID":"d8cce66671021be49acd4aba890d28bc","Type":"ContainerStarted","Data":"9f6877b2f1f0a1455db4c93c60d6f79f9eacea19b0713db33c19124b7d00f548"} Jul 22 15:04:59.197083 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.197053 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wxjcx" event={"ID":"ecbe0445-2b23-4b1b-b230-eb80fbb1915e","Type":"ContainerStarted","Data":"909c05861889e9b4ffff02eab623b83bbacd56472e4043d76e156b5fabdec3be"} Jul 22 15:04:59.223742 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.223695 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bjhfj" podStartSLOduration=4.11002703 podStartE2EDuration="24.223680821s" totalImagesPullingTime="20.113653791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:04:58.36822392 +0000 UTC m=+23.867123696" watchObservedRunningTime="2026-07-22 15:04:59.223680821 +0000 UTC m=+24.722580655" Jul 22 15:04:59.248005 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.247967 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-22.ec2.internal" podStartSLOduration=23.247955267 podStartE2EDuration="23.247955267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:04:59.223861608 +0000 UTC m=+24.722761387" watchObservedRunningTime="2026-07-22 15:04:59.247955267 +0000 UTC m=+24.746855043" Jul 22 15:04:59.248328 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.248294 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wxjcx" podStartSLOduration=4.124404768 podStartE2EDuration="24.24828671s" totalImagesPullingTime="20.123881942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:04:59.24821753 +0000 UTC m=+24.747117308" watchObservedRunningTime="2026-07-22 15:04:59.24828671 +0000 UTC m=+24.747186485" Jul 22 15:04:59.277268 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.277245 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b695k" Jul 22 15:04:59.277840 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.277821 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b695k" Jul 22 15:04:59.967405 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.967307 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T15:04:59.059205136Z","UUID":"cdc43080-eaf4-486d-9d73-d29c13534294","Handler":null,"Name":"","Endpoint":""} Jul 22 15:04:59.969220 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.969194 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 15:04:59.969333 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:04:59.969247 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 15:05:00.200452 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:00.200426 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b695k" Jul 22 15:05:00.201001 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:00.200643 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b695k" Jul 22 15:05:01.048165 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:01.048113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:05:01.048445 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:01.048172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:05:01.048445 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:01.048113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:05:01.048445 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:01.048242 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:05:01.048445 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:01.048354 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:05:01.048445 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:01.048425 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:05:01.204604 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:01.204570 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" event={"ID":"ccdb4fa1-83d8-4949-a402-394895fad8c6","Type":"ContainerStarted","Data":"69bd842de5d028740126d3e3f3420c4c12dc71c4d86cbdd37b693ea942dd101f"} Jul 22 15:05:01.208164 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:01.208136 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" event={"ID":"1734f3d9-8a01-419e-b81e-0ca3d0fbef10","Type":"ContainerStarted","Data":"ee6c8a10ef9573c80c7bcc088a4f4b42213e79c430e669c22c6cf1621ac6dfc2"} Jul 22 15:05:01.225731 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:01.225694 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7nsd" podStartSLOduration=3.658784046 podStartE2EDuration="26.225678931s" totalImagesPullingTime="22.566894885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:04:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:01.22553602 +0000 UTC m=+26.724435798" watchObservedRunningTime="2026-07-22 15:05:01.225678931 +0000 UTC m=+26.724578708" Jul 22 15:05:03.048513 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:03.048483 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:05:03.048513 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:03.048504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:05:03.049157 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:03.048489 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:05:03.049157 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:03.048594 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:05:03.049157 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:03.048675 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:05:03.049157 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:03.048755 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:05:03.215614 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:03.215529 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" event={"ID":"1734f3d9-8a01-419e-b81e-0ca3d0fbef10","Type":"ContainerStarted","Data":"a4768b22c1480795ebd6a34fa9795f34c697b16ec7221e9f15e896ed5580b8c2"} Jul 22 15:05:03.215806 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:03.215779 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:05:03.217376 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:03.217348 2580 generic.go:356] "Generic (PLEG): container finished" podID="719c1099-aeb4-4738-ba6f-ae117d6746b6" containerID="f2565ef87033112c335df6e44b17948c24776e09ddf892c0d8061173ee52556c" exitCode=0 Jul 22 15:05:03.217503 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:03.217384 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5trxb" event={"ID":"719c1099-aeb4-4738-ba6f-ae117d6746b6","Type":"ContainerDied","Data":"f2565ef87033112c335df6e44b17948c24776e09ddf892c0d8061173ee52556c"} Jul 22 15:05:03.231336 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:03.231320 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:05:03.252795 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:03.252760 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" podStartSLOduration=8.086137982 podStartE2EDuration="28.252749141s" totalImagesPullingTime="20.166611159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:03.252355759 +0000 UTC m=+28.751255553" watchObservedRunningTime="2026-07-22 15:05:03.252749141 +0000 UTC m=+28.751648917" Jul 22 15:05:04.220255 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:04.220041 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 15:05:04.220909 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:04.220232 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:05:04.236679 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:04.236643 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:05:04.544084 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:04.543990 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bjpkw"] Jul 22 15:05:04.544250 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:04.544101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:05:04.544250 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:04.544207 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:05:04.557568 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:04.557547 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-58ptc"] Jul 22 15:05:04.557674 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:04.557633 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:05:04.557711 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:04.557700 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:05:04.560791 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:04.560686 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bwrpd"] Jul 22 15:05:04.560791 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:04.560780 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:05:04.560934 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:04.560885 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:05:05.223856 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:05.223827 2580 generic.go:356] "Generic (PLEG): container finished" podID="719c1099-aeb4-4738-ba6f-ae117d6746b6" containerID="82ec416fe39edff2e16bb32a42b94e79f7bb3546dd071387c47fe880620adf4b" exitCode=0 Jul 22 15:05:05.224395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:05.223910 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5trxb" event={"ID":"719c1099-aeb4-4738-ba6f-ae117d6746b6","Type":"ContainerDied","Data":"82ec416fe39edff2e16bb32a42b94e79f7bb3546dd071387c47fe880620adf4b"} Jul 22 15:05:05.224395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:05.223954 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 15:05:06.048318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:06.048291 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:05:06.048473 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:06.048387 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:05:06.048524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:06.048394 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:05:06.048524 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:06.048474 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:05:06.048600 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:06.048387 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:05:06.048600 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:06.048576 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:05:06.225805 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:06.225776 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 15:05:06.481881 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:06.481848 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:05:07.231224 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:07.231197 2580 generic.go:356] "Generic (PLEG): container finished" podID="719c1099-aeb4-4738-ba6f-ae117d6746b6" containerID="f65b67790ebeabbafac327ce868334c2844833df5cd6377e4d6314aa37aa9a53" exitCode=0 Jul 22 15:05:07.231532 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:07.231288 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5trxb" event={"ID":"719c1099-aeb4-4738-ba6f-ae117d6746b6","Type":"ContainerDied","Data":"f65b67790ebeabbafac327ce868334c2844833df5cd6377e4d6314aa37aa9a53"} Jul 22 15:05:07.243024 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:07.242997 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" podUID="1734f3d9-8a01-419e-b81e-0ca3d0fbef10" containerName="ovnkube-controller" probeResult="failure" output="" Jul 22 15:05:08.048312 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:08.048275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:05:08.048495 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:08.048275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:05:08.048495 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:08.048405 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:05:08.048495 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:08.048291 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:05:08.048654 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:08.048488 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:05:08.048654 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:08.048593 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:05:08.642972 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:08.642934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:05:08.643472 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:08.643003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:05:08.643472 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:08.643103 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 15:05:08.643472 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:08.643156 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:05:08.643472 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:08.643203 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret podName:137d785b-18a1-41d5-9e16-d4c7f5f7af6b nodeName:}" failed. No retries permitted until 2026-07-22 15:05:40.643180371 +0000 UTC m=+66.142080139 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret") pod "global-pull-secret-syncer-bjpkw" (UID: "137d785b-18a1-41d5-9e16-d4c7f5f7af6b") : object "kube-system"/"original-pull-secret" not registered Jul 22 15:05:08.643472 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:08.643223 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs podName:5d3d0c39-a649-4919-aeb9-6566ca9c64f1 nodeName:}" failed. No retries permitted until 2026-07-22 15:05:40.64321426 +0000 UTC m=+66.142114016 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs") pod "network-metrics-daemon-bwrpd" (UID: "5d3d0c39-a649-4919-aeb9-6566ca9c64f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 15:05:08.743854 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:08.743807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpqq\" (UniqueName: \"kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq\") pod \"network-check-target-58ptc\" (UID: \"a9c3f4f1-b506-4b50-b350-a07d4a34cded\") " pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:05:08.744022 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:08.743993 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 15:05:08.744022 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:08.744020 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 15:05:08.744150 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:08.744035 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhpqq for pod openshift-network-diagnostics/network-check-target-58ptc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:05:08.744150 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:08.744100 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq podName:a9c3f4f1-b506-4b50-b350-a07d4a34cded nodeName:}" failed. No retries permitted until 2026-07-22 15:05:40.744079532 +0000 UTC m=+66.242979303 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhpqq" (UniqueName: "kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq") pod "network-check-target-58ptc" (UID: "a9c3f4f1-b506-4b50-b350-a07d4a34cded") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 15:05:10.048613 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.048374 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:05:10.049028 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.048408 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:05:10.049028 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:10.048720 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjpkw" podUID="137d785b-18a1-41d5-9e16-d4c7f5f7af6b" Jul 22 15:05:10.049028 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.048409 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:05:10.049028 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:10.048767 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58ptc" podUID="a9c3f4f1-b506-4b50-b350-a07d4a34cded" Jul 22 15:05:10.049028 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:10.048876 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwrpd" podUID="5d3d0c39-a649-4919-aeb9-6566ca9c64f1" Jul 22 15:05:10.296089 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.296018 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-22.ec2.internal" event="NodeReady" Jul 22 15:05:10.296269 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.296184 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 15:05:10.367382 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.367342 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vqrb5"] Jul 22 15:05:10.369986 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.369961 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2487l"] Jul 22 15:05:10.370112 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.370089 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:10.372032 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.372011 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2487l" Jul 22 15:05:10.373201 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.373179 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 15:05:10.373475 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.373457 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 15:05:10.373567 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.373512 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s4xt5\"" Jul 22 15:05:10.376201 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.376184 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 15:05:10.376673 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.376514 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gqg9k\"" Jul 22 15:05:10.376673 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.376562 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 15:05:10.376832 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.376760 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 15:05:10.383001 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.382974 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2487l"] Jul 22 15:05:10.390490 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.390438 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vqrb5"] Jul 22 15:05:10.457857 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.457828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65817246-491c-4aea-9ce0-2f582b22652f-cert\") pod \"ingress-canary-2487l\" (UID: \"65817246-491c-4aea-9ce0-2f582b22652f\") " pod="openshift-ingress-canary/ingress-canary-2487l" Jul 22 15:05:10.458011 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.457874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6bbg\" (UniqueName: \"kubernetes.io/projected/65817246-491c-4aea-9ce0-2f582b22652f-kube-api-access-x6bbg\") pod \"ingress-canary-2487l\" (UID: \"65817246-491c-4aea-9ce0-2f582b22652f\") " pod="openshift-ingress-canary/ingress-canary-2487l" Jul 22 15:05:10.458011 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.457896 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea197d01-92b2-44c6-87ac-7413a45fc26c-tmp-dir\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:10.458011 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.457985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvkbj\" (UniqueName: \"kubernetes.io/projected/ea197d01-92b2-44c6-87ac-7413a45fc26c-kube-api-access-pvkbj\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:10.458193 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.458018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:10.458193 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.458048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea197d01-92b2-44c6-87ac-7413a45fc26c-config-volume\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:10.559156 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.559104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6bbg\" (UniqueName: \"kubernetes.io/projected/65817246-491c-4aea-9ce0-2f582b22652f-kube-api-access-x6bbg\") pod \"ingress-canary-2487l\" (UID: \"65817246-491c-4aea-9ce0-2f582b22652f\") " pod="openshift-ingress-canary/ingress-canary-2487l" Jul 22 15:05:10.559310 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.559168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea197d01-92b2-44c6-87ac-7413a45fc26c-tmp-dir\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:10.559310 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.559210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvkbj\" (UniqueName: \"kubernetes.io/projected/ea197d01-92b2-44c6-87ac-7413a45fc26c-kube-api-access-pvkbj\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:10.559310 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.559236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:10.559310 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.559263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea197d01-92b2-44c6-87ac-7413a45fc26c-config-volume\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:10.559541 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.559347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65817246-491c-4aea-9ce0-2f582b22652f-cert\") pod \"ingress-canary-2487l\" (UID: \"65817246-491c-4aea-9ce0-2f582b22652f\") " pod="openshift-ingress-canary/ingress-canary-2487l" Jul 22 15:05:10.559541 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:10.559496 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 15:05:10.559635 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:10.559563 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls podName:ea197d01-92b2-44c6-87ac-7413a45fc26c nodeName:}" failed. No retries permitted until 2026-07-22 15:05:11.059541946 +0000 UTC m=+36.558441702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls") pod "dns-default-vqrb5" (UID: "ea197d01-92b2-44c6-87ac-7413a45fc26c") : secret "dns-default-metrics-tls" not found Jul 22 15:05:10.559635 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.559594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea197d01-92b2-44c6-87ac-7413a45fc26c-tmp-dir\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:10.559905 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.559885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea197d01-92b2-44c6-87ac-7413a45fc26c-config-volume\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:10.564180 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.564156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65817246-491c-4aea-9ce0-2f582b22652f-cert\") pod \"ingress-canary-2487l\" (UID: \"65817246-491c-4aea-9ce0-2f582b22652f\") " pod="openshift-ingress-canary/ingress-canary-2487l" Jul 22 15:05:10.571417 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.571396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvkbj\" (UniqueName: \"kubernetes.io/projected/ea197d01-92b2-44c6-87ac-7413a45fc26c-kube-api-access-pvkbj\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:10.571588 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.571572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6bbg\" (UniqueName: \"kubernetes.io/projected/65817246-491c-4aea-9ce0-2f582b22652f-kube-api-access-x6bbg\") pod \"ingress-canary-2487l\" (UID: \"65817246-491c-4aea-9ce0-2f582b22652f\") " pod="openshift-ingress-canary/ingress-canary-2487l" Jul 22 15:05:10.688191 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:10.688164 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2487l" Jul 22 15:05:11.062909 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:11.062884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:11.063323 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:11.063049 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 15:05:11.063323 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:11.063112 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls podName:ea197d01-92b2-44c6-87ac-7413a45fc26c nodeName:}" failed. No retries permitted until 2026-07-22 15:05:12.063097196 +0000 UTC m=+37.561996951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls") pod "dns-default-vqrb5" (UID: "ea197d01-92b2-44c6-87ac-7413a45fc26c") : secret "dns-default-metrics-tls" not found Jul 22 15:05:11.188701 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:11.188558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2487l"] Jul 22 15:05:11.192563 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:05:11.192538 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65817246_491c_4aea_9ce0_2f582b22652f.slice/crio-46e1d61b04f325fda9c395183a7e9fc42653a650d96cd4d445ec7409715a1c46 WatchSource:0}: Error finding container 46e1d61b04f325fda9c395183a7e9fc42653a650d96cd4d445ec7409715a1c46: Status 404 returned error can't find the container with id 46e1d61b04f325fda9c395183a7e9fc42653a650d96cd4d445ec7409715a1c46 Jul 22 15:05:11.244986 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:11.244957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5trxb" event={"ID":"719c1099-aeb4-4738-ba6f-ae117d6746b6","Type":"ContainerStarted","Data":"85d9b2b766c3b0e357885c839565f7efbbc7eb7654beacd85aa4435cbf9d1b69"} Jul 22 15:05:11.246128 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:11.246088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2487l" event={"ID":"65817246-491c-4aea-9ce0-2f582b22652f","Type":"ContainerStarted","Data":"46e1d61b04f325fda9c395183a7e9fc42653a650d96cd4d445ec7409715a1c46"} Jul 22 15:05:12.048315 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:12.048273 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:05:12.048487 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:12.048277 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:05:12.048487 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:12.048414 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:05:12.053647 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:12.053623 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 15:05:12.053647 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:12.053657 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 15:05:12.053825 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:12.053657 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9ldrl\"" Jul 22 15:05:12.053825 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:12.053657 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 15:05:12.054977 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:12.054949 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 15:05:12.055103 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:12.054987 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zbjvd\"" Jul 22 15:05:12.069814 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:12.069791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:12.070410 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:12.069898 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 15:05:12.070410 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:12.069954 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls podName:ea197d01-92b2-44c6-87ac-7413a45fc26c nodeName:}" failed. No retries permitted until 2026-07-22 15:05:14.069940671 +0000 UTC m=+39.568840425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls") pod "dns-default-vqrb5" (UID: "ea197d01-92b2-44c6-87ac-7413a45fc26c") : secret "dns-default-metrics-tls" not found Jul 22 15:05:12.251738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:12.251697 2580 generic.go:356] "Generic (PLEG): container finished" podID="719c1099-aeb4-4738-ba6f-ae117d6746b6" containerID="85d9b2b766c3b0e357885c839565f7efbbc7eb7654beacd85aa4435cbf9d1b69" exitCode=0 Jul 22 15:05:12.251738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:12.251729 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5trxb" event={"ID":"719c1099-aeb4-4738-ba6f-ae117d6746b6","Type":"ContainerDied","Data":"85d9b2b766c3b0e357885c839565f7efbbc7eb7654beacd85aa4435cbf9d1b69"} Jul 22 15:05:13.258218 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.258159 2580 generic.go:356] "Generic (PLEG): container finished" podID="719c1099-aeb4-4738-ba6f-ae117d6746b6" containerID="aec0f082aaff96b1b1b81c6801ae9a539dab2848d23bb7cd3769e96696b5f240" exitCode=0 Jul 22 15:05:13.258622 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.258245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5trxb" event={"ID":"719c1099-aeb4-4738-ba6f-ae117d6746b6","Type":"ContainerDied","Data":"aec0f082aaff96b1b1b81c6801ae9a539dab2848d23bb7cd3769e96696b5f240"} Jul 22 15:05:13.367986 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.367957 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-glwh9"] Jul 22 15:05:13.376554 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.376538 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-glwh9" Jul 22 15:05:13.380540 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.380521 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rlkwf\"" Jul 22 15:05:13.380540 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.380529 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 22 15:05:13.380687 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.380524 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 22 15:05:13.380687 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.380522 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 22 15:05:13.380881 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.380870 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 22 15:05:13.385827 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.385803 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-glwh9"] Jul 22 15:05:13.479954 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.479931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxx5g\" (UniqueName: \"kubernetes.io/projected/a7aa695b-d43e-479f-b279-ce1f14175abe-kube-api-access-dxx5g\") pod \"service-ca-58598695f7-glwh9\" (UID: \"a7aa695b-d43e-479f-b279-ce1f14175abe\") " pod="openshift-service-ca/service-ca-58598695f7-glwh9" Jul 22 15:05:13.480062 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.479963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a7aa695b-d43e-479f-b279-ce1f14175abe-signing-key\") pod \"service-ca-58598695f7-glwh9\" (UID: \"a7aa695b-d43e-479f-b279-ce1f14175abe\") " pod="openshift-service-ca/service-ca-58598695f7-glwh9" Jul 22 15:05:13.480062 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.480015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a7aa695b-d43e-479f-b279-ce1f14175abe-signing-cabundle\") pod \"service-ca-58598695f7-glwh9\" (UID: \"a7aa695b-d43e-479f-b279-ce1f14175abe\") " pod="openshift-service-ca/service-ca-58598695f7-glwh9" Jul 22 15:05:13.580890 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.580859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a7aa695b-d43e-479f-b279-ce1f14175abe-signing-key\") pod \"service-ca-58598695f7-glwh9\" (UID: \"a7aa695b-d43e-479f-b279-ce1f14175abe\") " pod="openshift-service-ca/service-ca-58598695f7-glwh9" Jul 22 15:05:13.580997 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.580943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a7aa695b-d43e-479f-b279-ce1f14175abe-signing-cabundle\") pod \"service-ca-58598695f7-glwh9\" (UID: \"a7aa695b-d43e-479f-b279-ce1f14175abe\") " pod="openshift-service-ca/service-ca-58598695f7-glwh9" Jul 22 15:05:13.581063 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.581035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxx5g\" (UniqueName: \"kubernetes.io/projected/a7aa695b-d43e-479f-b279-ce1f14175abe-kube-api-access-dxx5g\") pod \"service-ca-58598695f7-glwh9\" (UID: \"a7aa695b-d43e-479f-b279-ce1f14175abe\") " pod="openshift-service-ca/service-ca-58598695f7-glwh9" Jul 22 15:05:13.581780 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.581746 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a7aa695b-d43e-479f-b279-ce1f14175abe-signing-cabundle\") pod \"service-ca-58598695f7-glwh9\" (UID: \"a7aa695b-d43e-479f-b279-ce1f14175abe\") " pod="openshift-service-ca/service-ca-58598695f7-glwh9" Jul 22 15:05:13.584346 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.584326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a7aa695b-d43e-479f-b279-ce1f14175abe-signing-key\") pod \"service-ca-58598695f7-glwh9\" (UID: \"a7aa695b-d43e-479f-b279-ce1f14175abe\") " pod="openshift-service-ca/service-ca-58598695f7-glwh9" Jul 22 15:05:13.601179 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.601152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxx5g\" (UniqueName: \"kubernetes.io/projected/a7aa695b-d43e-479f-b279-ce1f14175abe-kube-api-access-dxx5g\") pod \"service-ca-58598695f7-glwh9\" (UID: \"a7aa695b-d43e-479f-b279-ce1f14175abe\") " pod="openshift-service-ca/service-ca-58598695f7-glwh9" Jul 22 15:05:13.686323 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.686292 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-glwh9" Jul 22 15:05:13.829948 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:13.827181 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-glwh9"] Jul 22 15:05:13.832096 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:05:13.832067 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7aa695b_d43e_479f_b279_ce1f14175abe.slice/crio-1312c5703e4c13e6ed4010dad040c6ab64f1bbd9cf11dbb7c09bbe1120391328 WatchSource:0}: Error finding container 1312c5703e4c13e6ed4010dad040c6ab64f1bbd9cf11dbb7c09bbe1120391328: Status 404 returned error can't find the container with id 1312c5703e4c13e6ed4010dad040c6ab64f1bbd9cf11dbb7c09bbe1120391328 Jul 22 15:05:14.086680 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.086611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:14.086789 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:14.086748 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 15:05:14.086842 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:14.086815 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls podName:ea197d01-92b2-44c6-87ac-7413a45fc26c nodeName:}" failed. No retries permitted until 2026-07-22 15:05:18.08679622 +0000 UTC m=+43.585695992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls") pod "dns-default-vqrb5" (UID: "ea197d01-92b2-44c6-87ac-7413a45fc26c") : secret "dns-default-metrics-tls" not found Jul 22 15:05:14.161957 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.161929 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-x8w7k"] Jul 22 15:05:14.193443 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.193417 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-x8w7k"] Jul 22 15:05:14.193564 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.193510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-x8w7k" Jul 22 15:05:14.198622 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.198602 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 22 15:05:14.199502 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.199472 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 22 15:05:14.199707 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.199692 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zpq6x\"" Jul 22 15:05:14.263876 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.263851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5trxb" event={"ID":"719c1099-aeb4-4738-ba6f-ae117d6746b6","Type":"ContainerStarted","Data":"addf7c9fc0e3f9af149b3a09c02edafbf1cbd017afc2ad900de3602a610fdde8"} Jul 22 15:05:14.265260 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.265242 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2487l" event={"ID":"65817246-491c-4aea-9ce0-2f582b22652f","Type":"ContainerStarted","Data":"a1b29038ae831a378c52bebd41f08756fa0c0bafddba94a066db492e23b3ea2e"} Jul 22 15:05:14.266300 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.266282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-glwh9" event={"ID":"a7aa695b-d43e-479f-b279-ce1f14175abe","Type":"ContainerStarted","Data":"1312c5703e4c13e6ed4010dad040c6ab64f1bbd9cf11dbb7c09bbe1120391328"} Jul 22 15:05:14.287651 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.287626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w85nm\" (UniqueName: \"kubernetes.io/projected/3c99be67-6a07-4e56-b299-4076b37cb720-kube-api-access-w85nm\") pod \"migrator-7cc9c7ffd7-x8w7k\" (UID: \"3c99be67-6a07-4e56-b299-4076b37cb720\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-x8w7k" Jul 22 15:05:14.307743 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.307706 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5trxb" podStartSLOduration=8.62041424 podStartE2EDuration="39.307693153s" totalImagesPullingTime="30.687278913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:04:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:14.307356746 +0000 UTC m=+39.806256522" watchObservedRunningTime="2026-07-22 15:05:14.307693153 +0000 UTC m=+39.806592929" Jul 22 15:05:14.388766 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.388697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w85nm\" (UniqueName: \"kubernetes.io/projected/3c99be67-6a07-4e56-b299-4076b37cb720-kube-api-access-w85nm\") pod \"migrator-7cc9c7ffd7-x8w7k\" (UID: \"3c99be67-6a07-4e56-b299-4076b37cb720\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-x8w7k" Jul 22 15:05:14.400193 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.400170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w85nm\" (UniqueName: \"kubernetes.io/projected/3c99be67-6a07-4e56-b299-4076b37cb720-kube-api-access-w85nm\") pod \"migrator-7cc9c7ffd7-x8w7k\" (UID: \"3c99be67-6a07-4e56-b299-4076b37cb720\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-x8w7k" Jul 22 15:05:14.502812 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.502783 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-x8w7k" Jul 22 15:05:14.681484 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.681399 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2487l" podStartSLOduration=2.337032165 podStartE2EDuration="4.681381203s" totalImagesPullingTime="2.344349038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:14.338301328 +0000 UTC m=+39.837201104" watchObservedRunningTime="2026-07-22 15:05:14.681381203 +0000 UTC m=+40.180280979" Jul 22 15:05:14.681938 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:14.681911 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-x8w7k"] Jul 22 15:05:14.686962 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:05:14.686934 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c99be67_6a07_4e56_b299_4076b37cb720.slice/crio-f0527876d99596ff32c384c234c0d7c09cbf8c43c5e6b682765addfed861c247 WatchSource:0}: Error finding container f0527876d99596ff32c384c234c0d7c09cbf8c43c5e6b682765addfed861c247: Status 404 returned error can't find the container with id f0527876d99596ff32c384c234c0d7c09cbf8c43c5e6b682765addfed861c247 Jul 22 15:05:15.270095 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:15.270061 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-x8w7k" event={"ID":"3c99be67-6a07-4e56-b299-4076b37cb720","Type":"ContainerStarted","Data":"f0527876d99596ff32c384c234c0d7c09cbf8c43c5e6b682765addfed861c247"} Jul 22 15:05:17.237967 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:17.237792 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t76c6_7003d3d9-bb04-433c-a3f4-a5e95f7f93da/dns-node-resolver/0.log" Jul 22 15:05:17.276673 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:17.276645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-x8w7k" event={"ID":"3c99be67-6a07-4e56-b299-4076b37cb720","Type":"ContainerStarted","Data":"4378af7285870fee5bee3a00202832354446f0a921ce98511365b3e12299cde4"} Jul 22 15:05:17.276799 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:17.276703 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-x8w7k" event={"ID":"3c99be67-6a07-4e56-b299-4076b37cb720","Type":"ContainerStarted","Data":"f71c5aa5ee9ea542aa28faef1160d3acadaf07e1682b6d2f05df6563c31c33df"} Jul 22 15:05:17.278064 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:17.278039 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-glwh9" event={"ID":"a7aa695b-d43e-479f-b279-ce1f14175abe","Type":"ContainerStarted","Data":"e692e04b8d0963eecd04f7e855f0d1a90c61799546c76f36a7523f774195cdc2"} Jul 22 15:05:17.296551 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:17.296510 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-x8w7k" podStartSLOduration=1.082322937 podStartE2EDuration="3.296498121s" totalImagesPullingTime="2.214175184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:05:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:17.295276455 +0000 UTC m=+42.794176241" watchObservedRunningTime="2026-07-22 15:05:17.296498121 +0000 UTC m=+42.795397897" Jul 22 15:05:17.314240 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:17.314201 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-glwh9" podStartSLOduration=1.246947159 podStartE2EDuration="4.314187496s" totalImagesPullingTime="3.067240337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:17.313670069 +0000 UTC m=+42.812569839" watchObservedRunningTime="2026-07-22 15:05:17.314187496 +0000 UTC m=+42.813087272" Jul 22 15:05:18.115932 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:18.115897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:18.116088 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:18.116039 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 15:05:18.116149 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:18.116097 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls podName:ea197d01-92b2-44c6-87ac-7413a45fc26c nodeName:}" failed. No retries permitted until 2026-07-22 15:05:26.116082371 +0000 UTC m=+51.614982130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls") pod "dns-default-vqrb5" (UID: "ea197d01-92b2-44c6-87ac-7413a45fc26c") : secret "dns-default-metrics-tls" not found Jul 22 15:05:18.436872 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:18.436800 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rrrth_e661d076-cf63-4136-9ac5-3afa121bda96/node-ca/0.log" Jul 22 15:05:18.836671 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:18.836648 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2487l_65817246-491c-4aea-9ce0-2f582b22652f/serve-healthcheck-canary/0.log" Jul 22 15:05:19.437910 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:19.437871 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-x8w7k_3c99be67-6a07-4e56-b299-4076b37cb720/migrator/0.log" Jul 22 15:05:19.635796 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:19.635766 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-x8w7k_3c99be67-6a07-4e56-b299-4076b37cb720/graceful-termination/0.log" Jul 22 15:05:26.172532 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:26.172502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:26.175312 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:26.175288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea197d01-92b2-44c6-87ac-7413a45fc26c-metrics-tls\") pod \"dns-default-vqrb5\" (UID: \"ea197d01-92b2-44c6-87ac-7413a45fc26c\") " pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:26.282448 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:26.282420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:26.420827 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:26.420799 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vqrb5"] Jul 22 15:05:27.310194 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:27.310153 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vqrb5" event={"ID":"ea197d01-92b2-44c6-87ac-7413a45fc26c","Type":"ContainerStarted","Data":"9da48f123ecc5de174f24aae2fd8f71b3a5df4e6488516a858ea5347152dbf0a"} Jul 22 15:05:29.318313 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:29.318278 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vqrb5" event={"ID":"ea197d01-92b2-44c6-87ac-7413a45fc26c","Type":"ContainerStarted","Data":"76f2fcdf810d9dae779ea5ea2b90533d98b07302d6c9978837710c0e36e20df4"} Jul 22 15:05:29.318756 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:29.318334 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vqrb5" event={"ID":"ea197d01-92b2-44c6-87ac-7413a45fc26c","Type":"ContainerStarted","Data":"19898e8684cce5f1622e0aa0b437455b31f9f41526c3e82696b9abe64a1307d0"} Jul 22 15:05:29.318756 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:29.318414 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:29.336377 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:29.336338 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vqrb5" podStartSLOduration=16.770915507 podStartE2EDuration="19.336323706s" totalImagesPullingTime="2.565408199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:29.335431378 +0000 UTC m=+54.834331153" watchObservedRunningTime="2026-07-22 15:05:29.336323706 +0000 UTC m=+54.835223483" Jul 22 15:05:36.020072 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.020037 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48"] Jul 22 15:05:36.044428 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.044401 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd"] Jul 22 15:05:36.044571 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.044519 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48" Jul 22 15:05:36.048225 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.048203 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 15:05:36.048491 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.048477 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-x7r6l\"" Jul 22 15:05:36.049786 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.049768 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 22 15:05:36.049873 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.049773 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 15:05:36.050083 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.050070 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 15:05:36.058093 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.058072 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48"] Jul 22 15:05:36.058222 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.058099 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd"] Jul 22 15:05:36.058284 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.058219 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.063797 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.063776 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 22 15:05:36.063913 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.063806 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 22 15:05:36.063974 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.063837 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 22 15:05:36.064272 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.064255 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 22 15:05:36.113095 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.113069 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qfxjl"] Jul 22 15:05:36.128707 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.128679 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.131885 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.131867 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t78n4\"" Jul 22 15:05:36.132137 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.132109 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 15:05:36.132342 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.132330 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 15:05:36.132570 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.132557 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 15:05:36.133134 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.133105 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 15:05:36.142315 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.142293 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qfxjl"] Jul 22 15:05:36.142527 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.142504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz66s\" (UniqueName: \"kubernetes.io/projected/5649fea3-e704-44b6-b6b4-876bc466bba4-kube-api-access-tz66s\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.142705 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.142684 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5649fea3-e704-44b6-b6b4-876bc466bba4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.142803 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.142712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.142803 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.142737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5649fea3-e704-44b6-b6b4-876bc466bba4-hub\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.142803 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.142794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef7358dd-fc63-494b-8dd4-9e2250b83a6a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cc5458858-srg48\" (UID: \"ef7358dd-fc63-494b-8dd4-9e2250b83a6a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48" Jul 22 15:05:36.142934 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.142838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-data-volume\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.142934 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.142859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5649fea3-e704-44b6-b6b4-876bc466bba4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.142934 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.142915 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsth7\" (UniqueName: \"kubernetes.io/projected/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-kube-api-access-vsth7\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.143040 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.142963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5649fea3-e704-44b6-b6b4-876bc466bba4-ca\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.143040 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.142985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-crio-socket\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.143040 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.143030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.143175 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.143056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649fea3-e704-44b6-b6b4-876bc466bba4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.143175 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.143076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-444f8\" (UniqueName: \"kubernetes.io/projected/ef7358dd-fc63-494b-8dd4-9e2250b83a6a-kube-api-access-444f8\") pod \"managed-serviceaccount-addon-agent-5cc5458858-srg48\" (UID: \"ef7358dd-fc63-494b-8dd4-9e2250b83a6a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48" Jul 22 15:05:36.243553 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.243510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5649fea3-e704-44b6-b6b4-876bc466bba4-ca\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.243553 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.243556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-crio-socket\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.243743 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.243630 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-crio-socket\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.243782 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.243737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.243820 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.243785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649fea3-e704-44b6-b6b4-876bc466bba4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.243820 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.243814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-444f8\" (UniqueName: \"kubernetes.io/projected/ef7358dd-fc63-494b-8dd4-9e2250b83a6a-kube-api-access-444f8\") pod \"managed-serviceaccount-addon-agent-5cc5458858-srg48\" (UID: \"ef7358dd-fc63-494b-8dd4-9e2250b83a6a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48" Jul 22 15:05:36.243901 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.243855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz66s\" (UniqueName: \"kubernetes.io/projected/5649fea3-e704-44b6-b6b4-876bc466bba4-kube-api-access-tz66s\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.244047 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.244023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5649fea3-e704-44b6-b6b4-876bc466bba4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.244163 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.244062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.244163 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.244093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5649fea3-e704-44b6-b6b4-876bc466bba4-hub\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.244163 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.244143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef7358dd-fc63-494b-8dd4-9e2250b83a6a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cc5458858-srg48\" (UID: \"ef7358dd-fc63-494b-8dd4-9e2250b83a6a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48" Jul 22 15:05:36.244315 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.244180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-data-volume\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.244315 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.244216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5649fea3-e704-44b6-b6b4-876bc466bba4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.244418 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.244398 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.244729 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.244606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-data-volume\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.244729 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.244630 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649fea3-e704-44b6-b6b4-876bc466bba4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.244729 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.244655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsth7\" (UniqueName: \"kubernetes.io/projected/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-kube-api-access-vsth7\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.246684 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.246602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5649fea3-e704-44b6-b6b4-876bc466bba4-ca\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.246985 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.246961 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.247106 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.247079 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5649fea3-e704-44b6-b6b4-876bc466bba4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.247236 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.247179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef7358dd-fc63-494b-8dd4-9e2250b83a6a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cc5458858-srg48\" (UID: \"ef7358dd-fc63-494b-8dd4-9e2250b83a6a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48" Jul 22 15:05:36.247599 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.247577 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5649fea3-e704-44b6-b6b4-876bc466bba4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.248133 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.248095 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5649fea3-e704-44b6-b6b4-876bc466bba4-hub\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.279273 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.279225 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-444f8\" (UniqueName: \"kubernetes.io/projected/ef7358dd-fc63-494b-8dd4-9e2250b83a6a-kube-api-access-444f8\") pod \"managed-serviceaccount-addon-agent-5cc5458858-srg48\" (UID: \"ef7358dd-fc63-494b-8dd4-9e2250b83a6a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48" Jul 22 15:05:36.322617 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.322595 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsth7\" (UniqueName: \"kubernetes.io/projected/4f42ba2c-c6f7-49e8-92bb-4ade149a922d-kube-api-access-vsth7\") pod \"insights-runtime-extractor-qfxjl\" (UID: \"4f42ba2c-c6f7-49e8-92bb-4ade149a922d\") " pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.322741 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.322644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz66s\" (UniqueName: \"kubernetes.io/projected/5649fea3-e704-44b6-b6b4-876bc466bba4-kube-api-access-tz66s\") pod \"cluster-proxy-proxy-agent-b7ddc47b9-fkvwd\" (UID: \"5649fea3-e704-44b6-b6b4-876bc466bba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.358554 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.358537 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48" Jul 22 15:05:36.366829 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.366810 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" Jul 22 15:05:36.454135 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.453916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qfxjl" Jul 22 15:05:36.509602 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.509575 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48"] Jul 22 15:05:36.514420 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:05:36.514388 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef7358dd_fc63_494b_8dd4_9e2250b83a6a.slice/crio-f0b5928d5bfd1fdd6505ba7994c29b01531771e6c98c9903302bf0e4d83db50c WatchSource:0}: Error finding container f0b5928d5bfd1fdd6505ba7994c29b01531771e6c98c9903302bf0e4d83db50c: Status 404 returned error can't find the container with id f0b5928d5bfd1fdd6505ba7994c29b01531771e6c98c9903302bf0e4d83db50c Jul 22 15:05:36.537829 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.537772 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd"] Jul 22 15:05:36.540943 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:05:36.540914 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5649fea3_e704_44b6_b6b4_876bc466bba4.slice/crio-898bbe447b509cd357e467015f8d8d6fbe7a3aa44a9ca4d0365c383adf532932 WatchSource:0}: Error finding container 898bbe447b509cd357e467015f8d8d6fbe7a3aa44a9ca4d0365c383adf532932: Status 404 returned error can't find the container with id 898bbe447b509cd357e467015f8d8d6fbe7a3aa44a9ca4d0365c383adf532932 Jul 22 15:05:36.595148 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:36.595107 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qfxjl"] Jul 22 15:05:36.597593 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:05:36.597570 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f42ba2c_c6f7_49e8_92bb_4ade149a922d.slice/crio-c3519c05fa9107f1983a180c919916a33668b2b0646f8870ad96f54873d791e9 WatchSource:0}: Error finding container c3519c05fa9107f1983a180c919916a33668b2b0646f8870ad96f54873d791e9: Status 404 returned error can't find the container with id c3519c05fa9107f1983a180c919916a33668b2b0646f8870ad96f54873d791e9 Jul 22 15:05:37.243509 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:37.243470 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvp4f" Jul 22 15:05:37.343541 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:37.343487 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" event={"ID":"5649fea3-e704-44b6-b6b4-876bc466bba4","Type":"ContainerStarted","Data":"898bbe447b509cd357e467015f8d8d6fbe7a3aa44a9ca4d0365c383adf532932"} Jul 22 15:05:37.345044 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:37.345009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qfxjl" event={"ID":"4f42ba2c-c6f7-49e8-92bb-4ade149a922d","Type":"ContainerStarted","Data":"c3519c05fa9107f1983a180c919916a33668b2b0646f8870ad96f54873d791e9"} Jul 22 15:05:37.346580 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:37.346488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48" event={"ID":"ef7358dd-fc63-494b-8dd4-9e2250b83a6a","Type":"ContainerStarted","Data":"f0b5928d5bfd1fdd6505ba7994c29b01531771e6c98c9903302bf0e4d83db50c"} Jul 22 15:05:39.324064 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:39.324037 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vqrb5" Jul 22 15:05:40.679414 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.679186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:05:40.679414 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.679253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:05:40.684166 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.683972 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 15:05:40.685475 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.685452 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 15:05:40.695660 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.695610 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/137d785b-18a1-41d5-9e16-d4c7f5f7af6b-original-pull-secret\") pod \"global-pull-secret-syncer-bjpkw\" (UID: \"137d785b-18a1-41d5-9e16-d4c7f5f7af6b\") " pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:05:40.696463 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.696444 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3d0c39-a649-4919-aeb9-6566ca9c64f1-metrics-certs\") pod \"network-metrics-daemon-bwrpd\" (UID: \"5d3d0c39-a649-4919-aeb9-6566ca9c64f1\") " pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:05:40.780278 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.780104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpqq\" (UniqueName: \"kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq\") pod \"network-check-target-58ptc\" (UID: \"a9c3f4f1-b506-4b50-b350-a07d4a34cded\") " pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:05:40.785345 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.785320 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 15:05:40.795673 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.795652 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 15:05:40.804849 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.804829 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhpqq\" (UniqueName: \"kubernetes.io/projected/a9c3f4f1-b506-4b50-b350-a07d4a34cded-kube-api-access-mhpqq\") pod \"network-check-target-58ptc\" (UID: \"a9c3f4f1-b506-4b50-b350-a07d4a34cded\") " pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:05:40.864059 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.863860 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9ldrl\"" Jul 22 15:05:40.871388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.871241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:05:40.871806 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.871788 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zbjvd\"" Jul 22 15:05:40.873493 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.873335 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjpkw" Jul 22 15:05:40.880158 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:40.879981 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwrpd" Jul 22 15:05:41.063279 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:41.063248 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-58ptc"] Jul 22 15:05:41.065822 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:05:41.065795 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c3f4f1_b506_4b50_b350_a07d4a34cded.slice/crio-a6049d79694247121790e990830adc172eb4957b90a4035baf08ed2667336d66 WatchSource:0}: Error finding container a6049d79694247121790e990830adc172eb4957b90a4035baf08ed2667336d66: Status 404 returned error can't find the container with id a6049d79694247121790e990830adc172eb4957b90a4035baf08ed2667336d66 Jul 22 15:05:41.278297 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:41.278225 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bjpkw"] Jul 22 15:05:41.280539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:41.280519 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bwrpd"] Jul 22 15:05:41.283466 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:05:41.283438 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137d785b_18a1_41d5_9e16_d4c7f5f7af6b.slice/crio-fcf70ef303aca9992fff72f8ac270309fb26ce537eb4ecc8e1fb3ca681b78764 WatchSource:0}: Error finding container fcf70ef303aca9992fff72f8ac270309fb26ce537eb4ecc8e1fb3ca681b78764: Status 404 returned error can't find the container with id fcf70ef303aca9992fff72f8ac270309fb26ce537eb4ecc8e1fb3ca681b78764 Jul 22 15:05:41.364973 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:41.364895 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48" event={"ID":"ef7358dd-fc63-494b-8dd4-9e2250b83a6a","Type":"ContainerStarted","Data":"a722b2bfa2d022131056c03ccdb6eea81513df5a7082252ce6f9d253d936d9de"} Jul 22 15:05:41.367179 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:41.367151 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" event={"ID":"5649fea3-e704-44b6-b6b4-876bc466bba4","Type":"ContainerStarted","Data":"18b8ff7d854f150688469beedb4146ed79f074cb59b832ec16a29d1267371427"} Jul 22 15:05:41.367283 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:41.367190 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" event={"ID":"5649fea3-e704-44b6-b6b4-876bc466bba4","Type":"ContainerStarted","Data":"49268fc4043ee9734c88b2e975837a1b733a1cc38dd6fe9e34ba1f660ea75df4"} Jul 22 15:05:41.367283 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:41.367209 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" event={"ID":"5649fea3-e704-44b6-b6b4-876bc466bba4","Type":"ContainerStarted","Data":"628bd573abf462877157205ab5e98d655969e94f2d75214c6dc873855c277cc7"} Jul 22 15:05:41.368474 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:41.368453 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bwrpd" event={"ID":"5d3d0c39-a649-4919-aeb9-6566ca9c64f1","Type":"ContainerStarted","Data":"7456c84f3c1649cab044078f7c6e1ac26152554f10e02561be158e94f956f62b"} Jul 22 15:05:41.369772 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:41.369746 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bjpkw" event={"ID":"137d785b-18a1-41d5-9e16-d4c7f5f7af6b","Type":"ContainerStarted","Data":"fcf70ef303aca9992fff72f8ac270309fb26ce537eb4ecc8e1fb3ca681b78764"} Jul 22 15:05:41.371014 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:41.370993 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-58ptc" event={"ID":"a9c3f4f1-b506-4b50-b350-a07d4a34cded","Type":"ContainerStarted","Data":"a6049d79694247121790e990830adc172eb4957b90a4035baf08ed2667336d66"} Jul 22 15:05:41.411977 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:41.411937 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc5458858-srg48" podStartSLOduration=2.479183309 podStartE2EDuration="6.411924671s" totalImagesPullingTime="3.932741362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:41.385272206 +0000 UTC m=+66.884171982" watchObservedRunningTime="2026-07-22 15:05:41.411924671 +0000 UTC m=+66.910824446" Jul 22 15:05:46.401342 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:46.401303 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bwrpd" event={"ID":"5d3d0c39-a649-4919-aeb9-6566ca9c64f1","Type":"ContainerStarted","Data":"db84c511f0f731db50fa2d49f402b84f0c8085e285a4eeb9b4bca164b1915702"} Jul 22 15:05:46.401846 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:46.401399 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bwrpd" event={"ID":"5d3d0c39-a649-4919-aeb9-6566ca9c64f1","Type":"ContainerStarted","Data":"4685a3dad72bf2bd924794fbca728bfbbbabdf6b778a40c3bf7fb9c961f4a4c4"} Jul 22 15:05:46.404739 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:46.404709 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qfxjl" event={"ID":"4f42ba2c-c6f7-49e8-92bb-4ade149a922d","Type":"ContainerStarted","Data":"74d41f778ff13da58ad4383570641ed023185f53275c9dde628734e1fcc74d3e"} Jul 22 15:05:46.422893 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:46.422842 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7ddc47b9-fkvwd" podStartSLOduration=7.505746852 podStartE2EDuration="11.422824214s" totalImagesPullingTime="3.917077362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:41.411890803 +0000 UTC m=+66.910790580" watchObservedRunningTime="2026-07-22 15:05:46.422824214 +0000 UTC m=+71.921723991" Jul 22 15:05:46.423499 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:46.423461 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bwrpd" podStartSLOduration=66.869846307 podStartE2EDuration="1m11.423449804s" totalImagesPullingTime="4.553603497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:46.422438699 +0000 UTC m=+71.921338476" watchObservedRunningTime="2026-07-22 15:05:46.423449804 +0000 UTC m=+71.922349580" Jul 22 15:05:51.425610 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:51.425529 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qfxjl" event={"ID":"4f42ba2c-c6f7-49e8-92bb-4ade149a922d","Type":"ContainerStarted","Data":"6db646e6a1e91dcdd2901a2f5b70f9ccc7b2c8988eb315c1dd6c3c2dcb958a7f"} Jul 22 15:05:51.426855 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:51.426824 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bjpkw" event={"ID":"137d785b-18a1-41d5-9e16-d4c7f5f7af6b","Type":"ContainerStarted","Data":"91de9874d52064b06f626c8be9bf7e700edc19eaea7be8ab4b75302ec2c54367"} Jul 22 15:05:51.428193 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:51.428173 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-58ptc" event={"ID":"a9c3f4f1-b506-4b50-b350-a07d4a34cded","Type":"ContainerStarted","Data":"5bffe1f81e9f88218a60ab2999900eb80154cec1e2ffb499f358cfbe2c9c7ad0"} Jul 22 15:05:51.428312 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:51.428302 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:05:51.449084 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:51.449047 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bjpkw" podStartSLOduration=66.579428384 podStartE2EDuration="1m16.449033886s" totalImagesPullingTime="9.869605502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:51.448656099 +0000 UTC m=+76.947555874" watchObservedRunningTime="2026-07-22 15:05:51.449033886 +0000 UTC m=+76.947933640" Jul 22 15:05:51.475619 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:51.475581 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-58ptc" podStartSLOduration=66.388126783 podStartE2EDuration="1m16.475567943s" totalImagesPullingTime="10.08744116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:51.475289304 +0000 UTC m=+76.974189080" watchObservedRunningTime="2026-07-22 15:05:51.475567943 +0000 UTC m=+76.974467752" Jul 22 15:05:57.450788 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:57.450753 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qfxjl" event={"ID":"4f42ba2c-c6f7-49e8-92bb-4ade149a922d","Type":"ContainerStarted","Data":"b64fbcd74304f1d15a49dc09d5e6dbfcc1b692950b6cebc17d53993a0af6f254"} Jul 22 15:05:57.476779 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:57.476737 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qfxjl" podStartSLOduration=1.631823367 podStartE2EDuration="21.476723477s" totalImagesPullingTime="19.84490011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:05:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:05:57.47561629 +0000 UTC m=+82.974516067" watchObservedRunningTime="2026-07-22 15:05:57.476723477 +0000 UTC m=+82.975623252" Jul 22 15:05:58.995160 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:58.995112 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cbmj7"] Jul 22 15:05:58.999995 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:58.999972 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.005579 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.005399 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 15:05:59.005579 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.005454 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 15:05:59.005579 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.005403 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 15:05:59.005579 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.005505 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mxpdq\"" Jul 22 15:05:59.005579 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.005468 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 15:05:59.005579 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.005460 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 15:05:59.005962 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.005803 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 15:05:59.114582 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.114555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.114766 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.114598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-tls\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.114766 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.114624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8857aff-0139-4cbc-b7c3-9761099e4db2-metrics-client-ca\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.114766 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.114655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-wtmp\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.114766 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.114683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8857aff-0139-4cbc-b7c3-9761099e4db2-sys\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.114766 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.114718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8857aff-0139-4cbc-b7c3-9761099e4db2-root\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.114999 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.114775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-accelerators-collector-config\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.114999 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.114836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxzxg\" (UniqueName: \"kubernetes.io/projected/a8857aff-0139-4cbc-b7c3-9761099e4db2-kube-api-access-bxzxg\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.114999 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.114866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-textfile\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.215967 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.215936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8857aff-0139-4cbc-b7c3-9761099e4db2-root\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216135 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.215976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-accelerators-collector-config\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216135 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.216021 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzxg\" (UniqueName: \"kubernetes.io/projected/a8857aff-0139-4cbc-b7c3-9761099e4db2-kube-api-access-bxzxg\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216135 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.216047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-textfile\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216135 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.216055 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8857aff-0139-4cbc-b7c3-9761099e4db2-root\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216135 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.216084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216419 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.216156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-tls\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216419 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.216188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8857aff-0139-4cbc-b7c3-9761099e4db2-metrics-client-ca\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216419 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.216218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-wtmp\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216419 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.216248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8857aff-0139-4cbc-b7c3-9761099e4db2-sys\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216419 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.216317 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8857aff-0139-4cbc-b7c3-9761099e4db2-sys\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216664 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:59.216422 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 15:05:59.216664 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.216434 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-wtmp\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216664 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:05:59.216489 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-tls podName:a8857aff-0139-4cbc-b7c3-9761099e4db2 nodeName:}" failed. No retries permitted until 2026-07-22 15:05:59.716469381 +0000 UTC m=+85.215369143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-tls") pod "node-exporter-cbmj7" (UID: "a8857aff-0139-4cbc-b7c3-9761099e4db2") : secret "node-exporter-tls" not found Jul 22 15:05:59.216664 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.216651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-accelerators-collector-config\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.216868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.216841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8857aff-0139-4cbc-b7c3-9761099e4db2-metrics-client-ca\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.217023 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.217005 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-textfile\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.218472 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.218452 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.247040 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.246996 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzxg\" (UniqueName: \"kubernetes.io/projected/a8857aff-0139-4cbc-b7c3-9761099e4db2-kube-api-access-bxzxg\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.720512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.720479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-tls\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.722788 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.722755 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8857aff-0139-4cbc-b7c3-9761099e4db2-node-exporter-tls\") pod \"node-exporter-cbmj7\" (UID: \"a8857aff-0139-4cbc-b7c3-9761099e4db2\") " pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.908927 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:05:59.908897 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cbmj7" Jul 22 15:05:59.920388 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:05:59.920359 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8857aff_0139_4cbc_b7c3_9761099e4db2.slice/crio-60d9ae35fa7cd4c9020d0db29b3e74f71ff170ae2ad062b20ee5553d9c8bcc0e WatchSource:0}: Error finding container 60d9ae35fa7cd4c9020d0db29b3e74f71ff170ae2ad062b20ee5553d9c8bcc0e: Status 404 returned error can't find the container with id 60d9ae35fa7cd4c9020d0db29b3e74f71ff170ae2ad062b20ee5553d9c8bcc0e Jul 22 15:06:00.461475 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:06:00.461441 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbmj7" event={"ID":"a8857aff-0139-4cbc-b7c3-9761099e4db2","Type":"ContainerStarted","Data":"60d9ae35fa7cd4c9020d0db29b3e74f71ff170ae2ad062b20ee5553d9c8bcc0e"} Jul 22 15:06:01.466455 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:06:01.466411 2580 generic.go:356] "Generic (PLEG): container finished" podID="a8857aff-0139-4cbc-b7c3-9761099e4db2" containerID="3e7856a1d79d66d8b0551859f1d1bb732a5d8305b9d34e6f027eb7546d589ac2" exitCode=0 Jul 22 15:06:01.466842 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:06:01.466500 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbmj7" event={"ID":"a8857aff-0139-4cbc-b7c3-9761099e4db2","Type":"ContainerDied","Data":"3e7856a1d79d66d8b0551859f1d1bb732a5d8305b9d34e6f027eb7546d589ac2"} Jul 22 15:06:02.471214 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:06:02.471185 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbmj7" event={"ID":"a8857aff-0139-4cbc-b7c3-9761099e4db2","Type":"ContainerStarted","Data":"242903ec28f0798592025ab213f57861495d7797907d8ead7b71b9fc9661a915"} Jul 22 15:06:02.471583 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:06:02.471234 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbmj7" event={"ID":"a8857aff-0139-4cbc-b7c3-9761099e4db2","Type":"ContainerStarted","Data":"ff395bda75fd5a46258dbfd410e02e10f43d804adef14648853c6f48fa66e5c3"} Jul 22 15:06:02.499226 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:06:02.499165 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cbmj7" podStartSLOduration=3.436826324 podStartE2EDuration="4.499149502s" totalImagesPullingTime="999.886278ms" totalInitContainerRuntime="62.4369ms" isStatefulPod=true podCreationTimestamp="2026-07-22 15:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:06:02.497184358 +0000 UTC m=+87.996084134" watchObservedRunningTime="2026-07-22 15:06:02.499149502 +0000 UTC m=+87.998049278" Jul 22 15:06:22.434090 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:06:22.434061 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-58ptc" Jul 22 15:09:08.235387 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.235039 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jcspj"] Jul 22 15:09:08.239056 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.238325 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jcspj" Jul 22 15:09:08.241278 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.241253 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 15:09:08.241381 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.241289 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 15:09:08.242436 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.242414 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-tn5ls\"" Jul 22 15:09:08.247140 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.247102 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jcspj"] Jul 22 15:09:08.256283 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.256262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbjdm\" (UniqueName: \"kubernetes.io/projected/287f4f1a-c9a7-4d8e-a72b-245911b07f82-kube-api-access-gbjdm\") pod \"cert-manager-55b4d954d5-jcspj\" (UID: \"287f4f1a-c9a7-4d8e-a72b-245911b07f82\") " pod="cert-manager/cert-manager-55b4d954d5-jcspj" Jul 22 15:09:08.256358 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.256314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/287f4f1a-c9a7-4d8e-a72b-245911b07f82-bound-sa-token\") pod \"cert-manager-55b4d954d5-jcspj\" (UID: \"287f4f1a-c9a7-4d8e-a72b-245911b07f82\") " pod="cert-manager/cert-manager-55b4d954d5-jcspj" Jul 22 15:09:08.357389 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.357361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/287f4f1a-c9a7-4d8e-a72b-245911b07f82-bound-sa-token\") pod \"cert-manager-55b4d954d5-jcspj\" (UID: \"287f4f1a-c9a7-4d8e-a72b-245911b07f82\") " pod="cert-manager/cert-manager-55b4d954d5-jcspj" Jul 22 15:09:08.357499 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.357400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbjdm\" (UniqueName: \"kubernetes.io/projected/287f4f1a-c9a7-4d8e-a72b-245911b07f82-kube-api-access-gbjdm\") pod \"cert-manager-55b4d954d5-jcspj\" (UID: \"287f4f1a-c9a7-4d8e-a72b-245911b07f82\") " pod="cert-manager/cert-manager-55b4d954d5-jcspj" Jul 22 15:09:08.366523 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.366498 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/287f4f1a-c9a7-4d8e-a72b-245911b07f82-bound-sa-token\") pod \"cert-manager-55b4d954d5-jcspj\" (UID: \"287f4f1a-c9a7-4d8e-a72b-245911b07f82\") " pod="cert-manager/cert-manager-55b4d954d5-jcspj" Jul 22 15:09:08.366821 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.366801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbjdm\" (UniqueName: \"kubernetes.io/projected/287f4f1a-c9a7-4d8e-a72b-245911b07f82-kube-api-access-gbjdm\") pod \"cert-manager-55b4d954d5-jcspj\" (UID: \"287f4f1a-c9a7-4d8e-a72b-245911b07f82\") " pod="cert-manager/cert-manager-55b4d954d5-jcspj" Jul 22 15:09:08.547579 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.547496 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jcspj" Jul 22 15:09:08.669675 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:08.669643 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jcspj"] Jul 22 15:09:08.673354 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:09:08.673322 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod287f4f1a_c9a7_4d8e_a72b_245911b07f82.slice/crio-d17e67eaec143a8dd333721d7022edaa2e3eca822311dec39bac542cc4e81f52 WatchSource:0}: Error finding container d17e67eaec143a8dd333721d7022edaa2e3eca822311dec39bac542cc4e81f52: Status 404 returned error can't find the container with id d17e67eaec143a8dd333721d7022edaa2e3eca822311dec39bac542cc4e81f52 Jul 22 15:09:09.084942 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:09.084908 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jcspj" event={"ID":"287f4f1a-c9a7-4d8e-a72b-245911b07f82","Type":"ContainerStarted","Data":"d17e67eaec143a8dd333721d7022edaa2e3eca822311dec39bac542cc4e81f52"} Jul 22 15:09:12.098096 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:12.098060 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jcspj" event={"ID":"287f4f1a-c9a7-4d8e-a72b-245911b07f82","Type":"ContainerStarted","Data":"506b817d156b9c416901cae99aaa55d4699c4f7b9ee345d2869bc9d143d95dc3"} Jul 22 15:09:34.888131 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:34.888085 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 15:09:37.724359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.724306 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jcspj" podStartSLOduration=26.595832383 podStartE2EDuration="29.724290881s" totalImagesPullingTime="3.128458498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:09:12.118435464 +0000 UTC m=+277.617335241" watchObservedRunningTime="2026-07-22 15:09:37.724290881 +0000 UTC m=+303.223190657" Jul 22 15:09:37.725735 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.724612 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z"] Jul 22 15:09:37.726480 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.726465 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.731075 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.731048 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-klfb9\"" Jul 22 15:09:37.731220 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.731093 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 15:09:37.731220 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.731048 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 15:09:37.731220 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.731145 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 15:09:37.731220 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.731171 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 15:09:37.749425 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.749404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a8f4ced-8d46-49b5-bd5e-89a35811254f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.749522 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.749433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a8f4ced-8d46-49b5-bd5e-89a35811254f-manifests\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.749522 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.749457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvng9\" (UniqueName: \"kubernetes.io/projected/0a8f4ced-8d46-49b5-bd5e-89a35811254f-kube-api-access-cvng9\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.749598 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.749566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a8f4ced-8d46-49b5-bd5e-89a35811254f-tmp\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.749598 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.749588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a8f4ced-8d46-49b5-bd5e-89a35811254f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.752228 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.752205 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z"] Jul 22 15:09:37.850581 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.850555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a8f4ced-8d46-49b5-bd5e-89a35811254f-tmp\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.850581 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.850581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a8f4ced-8d46-49b5-bd5e-89a35811254f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.850752 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.850608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a8f4ced-8d46-49b5-bd5e-89a35811254f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.850752 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.850724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a8f4ced-8d46-49b5-bd5e-89a35811254f-manifests\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.850854 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.850761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvng9\" (UniqueName: \"kubernetes.io/projected/0a8f4ced-8d46-49b5-bd5e-89a35811254f-kube-api-access-cvng9\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.851138 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.851098 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a8f4ced-8d46-49b5-bd5e-89a35811254f-manifests\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.852975 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.852956 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a8f4ced-8d46-49b5-bd5e-89a35811254f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.853596 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.853572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a8f4ced-8d46-49b5-bd5e-89a35811254f-tmp\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.853777 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.853761 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a8f4ced-8d46-49b5-bd5e-89a35811254f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:37.861533 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:37.861510 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvng9\" (UniqueName: \"kubernetes.io/projected/0a8f4ced-8d46-49b5-bd5e-89a35811254f-kube-api-access-cvng9\") pod \"opendatahub-operator-controller-manager-7777b6b677-t2j4z\" (UID: \"0a8f4ced-8d46-49b5-bd5e-89a35811254f\") " pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:38.035974 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:38.035900 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:38.235109 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:38.235073 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z"] Jul 22 15:09:38.237249 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:09:38.237224 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8f4ced_8d46_49b5_bd5e_89a35811254f.slice/crio-d8360122b28a6bbe6b9b9b632141eacc1eaa8203ed7b7267f827dad21a53a204 WatchSource:0}: Error finding container d8360122b28a6bbe6b9b9b632141eacc1eaa8203ed7b7267f827dad21a53a204: Status 404 returned error can't find the container with id d8360122b28a6bbe6b9b9b632141eacc1eaa8203ed7b7267f827dad21a53a204 Jul 22 15:09:38.238992 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:38.238975 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 15:09:39.191885 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:39.191846 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" event={"ID":"0a8f4ced-8d46-49b5-bd5e-89a35811254f","Type":"ContainerStarted","Data":"d8360122b28a6bbe6b9b9b632141eacc1eaa8203ed7b7267f827dad21a53a204"} Jul 22 15:09:42.203137 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:42.203091 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" event={"ID":"0a8f4ced-8d46-49b5-bd5e-89a35811254f","Type":"ContainerStarted","Data":"c8273b0049c89bf64727c364b534a3cc622a997dbf04193443f45418e6c2d665"} Jul 22 15:09:43.207206 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:43.207171 2580 generic.go:356] "Generic (PLEG): container finished" podID="0a8f4ced-8d46-49b5-bd5e-89a35811254f" containerID="c8273b0049c89bf64727c364b534a3cc622a997dbf04193443f45418e6c2d665" exitCode=0 Jul 22 15:09:43.207693 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:43.207225 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" event={"ID":"0a8f4ced-8d46-49b5-bd5e-89a35811254f","Type":"ContainerDied","Data":"c8273b0049c89bf64727c364b534a3cc622a997dbf04193443f45418e6c2d665"} Jul 22 15:09:44.212588 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:44.212553 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" event={"ID":"0a8f4ced-8d46-49b5-bd5e-89a35811254f","Type":"ContainerStarted","Data":"f16f7a7e0e7090b4cd2300b5799d40f0c7a1cafb191656d023b96799316f5bbe"} Jul 22 15:09:44.212984 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:44.212672 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:44.385613 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:44.385568 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" podStartSLOduration=3.053729256 podStartE2EDuration="7.385552227s" totalImagesPullingTime="3.306530031s" totalInitContainerRuntime="1.02529294s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:09:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:09:44.307173699 +0000 UTC m=+309.806073475" watchObservedRunningTime="2026-07-22 15:09:44.385552227 +0000 UTC m=+309.884452002" Jul 22 15:09:55.217769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:55.217742 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7777b6b677-t2j4z" Jul 22 15:09:58.389893 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.389828 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59659ff488-rp2tq"] Jul 22 15:09:58.393007 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.392987 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" Jul 22 15:09:58.397271 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.397252 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 15:09:58.398595 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.398577 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-npzhb\"" Jul 22 15:09:58.398687 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.398595 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 15:09:58.398687 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.398605 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 15:09:58.398687 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.398635 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 15:09:58.417865 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.417841 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59659ff488-rp2tq"] Jul 22 15:09:58.493434 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.493410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb85daf-a839-45f0-bf23-24cfc9ed2f06-tls-certs\") pod \"kube-auth-proxy-59659ff488-rp2tq\" (UID: \"0fb85daf-a839-45f0-bf23-24cfc9ed2f06\") " pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" Jul 22 15:09:58.493533 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.493456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fb85daf-a839-45f0-bf23-24cfc9ed2f06-tmp\") pod \"kube-auth-proxy-59659ff488-rp2tq\" (UID: \"0fb85daf-a839-45f0-bf23-24cfc9ed2f06\") " pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" Jul 22 15:09:58.493533 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.493480 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcftn\" (UniqueName: \"kubernetes.io/projected/0fb85daf-a839-45f0-bf23-24cfc9ed2f06-kube-api-access-hcftn\") pod \"kube-auth-proxy-59659ff488-rp2tq\" (UID: \"0fb85daf-a839-45f0-bf23-24cfc9ed2f06\") " pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" Jul 22 15:09:58.593932 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.593893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb85daf-a839-45f0-bf23-24cfc9ed2f06-tls-certs\") pod \"kube-auth-proxy-59659ff488-rp2tq\" (UID: \"0fb85daf-a839-45f0-bf23-24cfc9ed2f06\") " pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" Jul 22 15:09:58.594029 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.593949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fb85daf-a839-45f0-bf23-24cfc9ed2f06-tmp\") pod \"kube-auth-proxy-59659ff488-rp2tq\" (UID: \"0fb85daf-a839-45f0-bf23-24cfc9ed2f06\") " pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" Jul 22 15:09:58.594029 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.593975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcftn\" (UniqueName: \"kubernetes.io/projected/0fb85daf-a839-45f0-bf23-24cfc9ed2f06-kube-api-access-hcftn\") pod \"kube-auth-proxy-59659ff488-rp2tq\" (UID: \"0fb85daf-a839-45f0-bf23-24cfc9ed2f06\") " pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" Jul 22 15:09:58.596216 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.596191 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fb85daf-a839-45f0-bf23-24cfc9ed2f06-tmp\") pod \"kube-auth-proxy-59659ff488-rp2tq\" (UID: \"0fb85daf-a839-45f0-bf23-24cfc9ed2f06\") " pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" Jul 22 15:09:58.596417 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.596400 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb85daf-a839-45f0-bf23-24cfc9ed2f06-tls-certs\") pod \"kube-auth-proxy-59659ff488-rp2tq\" (UID: \"0fb85daf-a839-45f0-bf23-24cfc9ed2f06\") " pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" Jul 22 15:09:58.604983 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.604964 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcftn\" (UniqueName: \"kubernetes.io/projected/0fb85daf-a839-45f0-bf23-24cfc9ed2f06-kube-api-access-hcftn\") pod \"kube-auth-proxy-59659ff488-rp2tq\" (UID: \"0fb85daf-a839-45f0-bf23-24cfc9ed2f06\") " pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" Jul 22 15:09:58.666754 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.666702 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ptwkj"] Jul 22 15:09:58.669720 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.669706 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" Jul 22 15:09:58.673329 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.673306 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 22 15:09:58.673422 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.673406 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-xflq9\"" Jul 22 15:09:58.699602 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.699577 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ptwkj"] Jul 22 15:09:58.701892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.701876 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" Jul 22 15:09:58.795687 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.795645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sffcq\" (UniqueName: \"kubernetes.io/projected/0a5d6422-e617-4392-bb78-ea9f33304047-kube-api-access-sffcq\") pod \"odh-model-controller-7998c67dc4-ptwkj\" (UID: \"0a5d6422-e617-4392-bb78-ea9f33304047\") " pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" Jul 22 15:09:58.795826 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.795689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a5d6422-e617-4392-bb78-ea9f33304047-cert\") pod \"odh-model-controller-7998c67dc4-ptwkj\" (UID: \"0a5d6422-e617-4392-bb78-ea9f33304047\") " pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" Jul 22 15:09:58.861165 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.861135 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59659ff488-rp2tq"] Jul 22 15:09:58.862801 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:09:58.862776 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb85daf_a839_45f0_bf23_24cfc9ed2f06.slice/crio-c9f4857ac05c75759f2e241f4edecc1152dc5e16a86260404ab69321400864aa WatchSource:0}: Error finding container c9f4857ac05c75759f2e241f4edecc1152dc5e16a86260404ab69321400864aa: Status 404 returned error can't find the container with id c9f4857ac05c75759f2e241f4edecc1152dc5e16a86260404ab69321400864aa Jul 22 15:09:58.896610 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.896586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffcq\" (UniqueName: \"kubernetes.io/projected/0a5d6422-e617-4392-bb78-ea9f33304047-kube-api-access-sffcq\") pod \"odh-model-controller-7998c67dc4-ptwkj\" (UID: \"0a5d6422-e617-4392-bb78-ea9f33304047\") " pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" Jul 22 15:09:58.896712 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.896620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a5d6422-e617-4392-bb78-ea9f33304047-cert\") pod \"odh-model-controller-7998c67dc4-ptwkj\" (UID: \"0a5d6422-e617-4392-bb78-ea9f33304047\") " pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" Jul 22 15:09:58.896777 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:09:58.896722 2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 22 15:09:58.896777 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:09:58.896772 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a5d6422-e617-4392-bb78-ea9f33304047-cert podName:0a5d6422-e617-4392-bb78-ea9f33304047 nodeName:}" failed. No retries permitted until 2026-07-22 15:09:59.396755204 +0000 UTC m=+324.895654958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0a5d6422-e617-4392-bb78-ea9f33304047-cert") pod "odh-model-controller-7998c67dc4-ptwkj" (UID: "0a5d6422-e617-4392-bb78-ea9f33304047") : secret "odh-model-controller-webhook-cert" not found Jul 22 15:09:58.920765 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:58.920709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffcq\" (UniqueName: \"kubernetes.io/projected/0a5d6422-e617-4392-bb78-ea9f33304047-kube-api-access-sffcq\") pod \"odh-model-controller-7998c67dc4-ptwkj\" (UID: \"0a5d6422-e617-4392-bb78-ea9f33304047\") " pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" Jul 22 15:09:59.262320 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.262239 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" event={"ID":"0fb85daf-a839-45f0-bf23-24cfc9ed2f06","Type":"ContainerStarted","Data":"c9f4857ac05c75759f2e241f4edecc1152dc5e16a86260404ab69321400864aa"} Jul 22 15:09:59.400389 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.400357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a5d6422-e617-4392-bb78-ea9f33304047-cert\") pod \"odh-model-controller-7998c67dc4-ptwkj\" (UID: \"0a5d6422-e617-4392-bb78-ea9f33304047\") " pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" Jul 22 15:09:59.402671 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.402652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a5d6422-e617-4392-bb78-ea9f33304047-cert\") pod \"odh-model-controller-7998c67dc4-ptwkj\" (UID: \"0a5d6422-e617-4392-bb78-ea9f33304047\") " pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" Jul 22 15:09:59.579109 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.579083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" Jul 22 15:09:59.736819 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.736789 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bz9tp"] Jul 22 15:09:59.740208 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.740172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" Jul 22 15:09:59.740906 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.740884 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ptwkj"] Jul 22 15:09:59.744802 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:09:59.744777 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5d6422_e617_4392_bb78_ea9f33304047.slice/crio-7e554de229ad97ae1dece8b374e4eb966176793b1f0a4f751f9b02df77ed594a WatchSource:0}: Error finding container 7e554de229ad97ae1dece8b374e4eb966176793b1f0a4f751f9b02df77ed594a: Status 404 returned error can't find the container with id 7e554de229ad97ae1dece8b374e4eb966176793b1f0a4f751f9b02df77ed594a Jul 22 15:09:59.753409 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.753308 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 22 15:09:59.770698 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.770677 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bz9tp"] Jul 22 15:09:59.804516 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.804490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8-tls-certs\") pod \"model-serving-api-54d89bd5fc-bz9tp\" (UID: \"f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8\") " pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" Jul 22 15:09:59.804619 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.804535 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwmt\" (UniqueName: \"kubernetes.io/projected/f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8-kube-api-access-fmwmt\") pod \"model-serving-api-54d89bd5fc-bz9tp\" (UID: \"f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8\") " pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" Jul 22 15:09:59.904978 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.904892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwmt\" (UniqueName: \"kubernetes.io/projected/f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8-kube-api-access-fmwmt\") pod \"model-serving-api-54d89bd5fc-bz9tp\" (UID: \"f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8\") " pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" Jul 22 15:09:59.904978 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.904973 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8-tls-certs\") pod \"model-serving-api-54d89bd5fc-bz9tp\" (UID: \"f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8\") " pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" Jul 22 15:09:59.907698 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.907667 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8-tls-certs\") pod \"model-serving-api-54d89bd5fc-bz9tp\" (UID: \"f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8\") " pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" Jul 22 15:09:59.914973 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:09:59.914949 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwmt\" (UniqueName: \"kubernetes.io/projected/f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8-kube-api-access-fmwmt\") pod \"model-serving-api-54d89bd5fc-bz9tp\" (UID: \"f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8\") " pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" Jul 22 15:10:00.059923 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:00.059887 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" Jul 22 15:10:00.215347 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:00.215276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bz9tp"] Jul 22 15:10:00.218153 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:10:00.218092 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d40f5b_9a8a_4650_b1a2_51ba2a866fe8.slice/crio-074b0cf2cf2890769f62b97a414f9ece5d3e6280bae127e5165915716fa99ec4 WatchSource:0}: Error finding container 074b0cf2cf2890769f62b97a414f9ece5d3e6280bae127e5165915716fa99ec4: Status 404 returned error can't find the container with id 074b0cf2cf2890769f62b97a414f9ece5d3e6280bae127e5165915716fa99ec4 Jul 22 15:10:00.266960 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:00.266930 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" event={"ID":"f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8","Type":"ContainerStarted","Data":"074b0cf2cf2890769f62b97a414f9ece5d3e6280bae127e5165915716fa99ec4"} Jul 22 15:10:00.268031 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:00.268009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" event={"ID":"0a5d6422-e617-4392-bb78-ea9f33304047","Type":"ContainerStarted","Data":"7e554de229ad97ae1dece8b374e4eb966176793b1f0a4f751f9b02df77ed594a"} Jul 22 15:10:04.285820 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:04.285789 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" event={"ID":"0fb85daf-a839-45f0-bf23-24cfc9ed2f06","Type":"ContainerStarted","Data":"f296c0860e973b498203ea44fc586dbf0ade0d08c477dfb5d38218732751e102"} Jul 22 15:10:04.287200 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:04.287175 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" event={"ID":"f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8","Type":"ContainerStarted","Data":"0fb7ee097f1591906961be91d1debaaeff11812043e5e80b647855f9588d6b22"} Jul 22 15:10:04.287324 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:04.287284 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" Jul 22 15:10:04.288593 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:04.288574 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" event={"ID":"0a5d6422-e617-4392-bb78-ea9f33304047","Type":"ContainerStarted","Data":"392a9b5401845fd7d7088249f647734d126a5375475b335919d0aed11bee2040"} Jul 22 15:10:04.288707 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:04.288697 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" Jul 22 15:10:04.305679 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:04.305630 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59659ff488-rp2tq" podStartSLOduration=1.8898452909999999 podStartE2EDuration="6.305618116s" totalImagesPullingTime="4.415772825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:10:04.304613898 +0000 UTC m=+329.803513674" watchObservedRunningTime="2026-07-22 15:10:04.305618116 +0000 UTC m=+329.804517892" Jul 22 15:10:04.322427 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:04.322387 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" podStartSLOduration=2.778360543 podStartE2EDuration="6.322373186s" totalImagesPullingTime="3.544012643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:10:04.320881156 +0000 UTC m=+329.819780934" watchObservedRunningTime="2026-07-22 15:10:04.322373186 +0000 UTC m=+329.821272963" Jul 22 15:10:04.348483 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:04.348444 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" podStartSLOduration=2.287643748 podStartE2EDuration="5.348430518s" totalImagesPullingTime="3.06078677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:10:04.347360037 +0000 UTC m=+329.846259812" watchObservedRunningTime="2026-07-22 15:10:04.348430518 +0000 UTC m=+329.847330294" Jul 22 15:10:06.608924 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.608889 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp"] Jul 22 15:10:06.612308 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.612286 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.622224 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.622203 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 15:10:06.622340 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.622209 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 15:10:06.622666 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.622648 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qvl7d\"" Jul 22 15:10:06.622758 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.622716 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 15:10:06.622758 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.622753 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 15:10:06.630504 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.630483 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp"] Jul 22 15:10:06.763835 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.763798 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7734426c-3de3-4d5c-9bf4-ce2860bade98-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.763987 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.763839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7734426c-3de3-4d5c-9bf4-ce2860bade98-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.763987 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.763870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdtrc\" (UniqueName: \"kubernetes.io/projected/7734426c-3de3-4d5c-9bf4-ce2860bade98-kube-api-access-zdtrc\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.764070 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.763980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7734426c-3de3-4d5c-9bf4-ce2860bade98-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.764070 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.764018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7734426c-3de3-4d5c-9bf4-ce2860bade98-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.764070 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.764049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7734426c-3de3-4d5c-9bf4-ce2860bade98-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.764193 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.764105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7734426c-3de3-4d5c-9bf4-ce2860bade98-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.865403 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.865330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7734426c-3de3-4d5c-9bf4-ce2860bade98-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.865403 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.865367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdtrc\" (UniqueName: \"kubernetes.io/projected/7734426c-3de3-4d5c-9bf4-ce2860bade98-kube-api-access-zdtrc\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.865611 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.865497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7734426c-3de3-4d5c-9bf4-ce2860bade98-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.865611 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.865533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7734426c-3de3-4d5c-9bf4-ce2860bade98-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.865611 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.865552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7734426c-3de3-4d5c-9bf4-ce2860bade98-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.865766 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.865690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7734426c-3de3-4d5c-9bf4-ce2860bade98-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.865766 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.865725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7734426c-3de3-4d5c-9bf4-ce2860bade98-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.866413 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.866390 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7734426c-3de3-4d5c-9bf4-ce2860bade98-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.867954 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.867927 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7734426c-3de3-4d5c-9bf4-ce2860bade98-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.868053 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.867975 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7734426c-3de3-4d5c-9bf4-ce2860bade98-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.868214 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.868197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7734426c-3de3-4d5c-9bf4-ce2860bade98-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.868281 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.868220 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7734426c-3de3-4d5c-9bf4-ce2860bade98-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.888068 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.888048 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7734426c-3de3-4d5c-9bf4-ce2860bade98-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.888470 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.888447 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdtrc\" (UniqueName: \"kubernetes.io/projected/7734426c-3de3-4d5c-9bf4-ce2860bade98-kube-api-access-zdtrc\") pod \"istiod-openshift-gateway-94bb8fbfd-nxctp\" (UID: \"7734426c-3de3-4d5c-9bf4-ce2860bade98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:06.922351 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:06.922333 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:07.061202 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:07.061172 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp"] Jul 22 15:10:07.063623 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:10:07.063584 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7734426c_3de3_4d5c_9bf4_ce2860bade98.slice/crio-0a839aad85e61013ce7d0b99c90fa1b4e41aebed1c54b73267ef9fbd64dd28f4 WatchSource:0}: Error finding container 0a839aad85e61013ce7d0b99c90fa1b4e41aebed1c54b73267ef9fbd64dd28f4: Status 404 returned error can't find the container with id 0a839aad85e61013ce7d0b99c90fa1b4e41aebed1c54b73267ef9fbd64dd28f4 Jul 22 15:10:07.300190 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:07.300098 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" event={"ID":"7734426c-3de3-4d5c-9bf4-ce2860bade98","Type":"ContainerStarted","Data":"0a839aad85e61013ce7d0b99c90fa1b4e41aebed1c54b73267ef9fbd64dd28f4"} Jul 22 15:10:09.477770 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:09.477725 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:10:09.478066 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:09.477799 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:10:10.178024 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.177989 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf"] Jul 22 15:10:10.181937 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.181911 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" Jul 22 15:10:10.185290 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.185266 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 22 15:10:10.185982 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.185958 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-b4q9r\"" Jul 22 15:10:10.200081 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.200024 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf"] Jul 22 15:10:10.293188 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.293158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j5xq\" (UniqueName: \"kubernetes.io/projected/b202d9b6-1e29-4995-aa00-251fdf0b356b-kube-api-access-2j5xq\") pod \"llmisvc-controller-manager-777c6bd7d5-wrlvf\" (UID: \"b202d9b6-1e29-4995-aa00-251fdf0b356b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" Jul 22 15:10:10.293356 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.293203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b202d9b6-1e29-4995-aa00-251fdf0b356b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wrlvf\" (UID: \"b202d9b6-1e29-4995-aa00-251fdf0b356b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" Jul 22 15:10:10.315434 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.315407 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" event={"ID":"7734426c-3de3-4d5c-9bf4-ce2860bade98","Type":"ContainerStarted","Data":"8f50af5d39c8bb16812bfceda09795b5650178e3bfc7f0b9d7627ff2c016feca"} Jul 22 15:10:10.315615 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.315598 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:10.317275 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.317253 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-nxctp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 15:10:10.317367 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.317300 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" podUID="7734426c-3de3-4d5c-9bf4-ce2860bade98" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 15:10:10.344399 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.344354 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" podStartSLOduration=1.932475981 podStartE2EDuration="4.344336152s" totalImagesPullingTime="2.411860171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:10:10.341863381 +0000 UTC m=+335.840763157" watchObservedRunningTime="2026-07-22 15:10:10.344336152 +0000 UTC m=+335.843235925" Jul 22 15:10:10.393896 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.393873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j5xq\" (UniqueName: \"kubernetes.io/projected/b202d9b6-1e29-4995-aa00-251fdf0b356b-kube-api-access-2j5xq\") pod \"llmisvc-controller-manager-777c6bd7d5-wrlvf\" (UID: \"b202d9b6-1e29-4995-aa00-251fdf0b356b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" Jul 22 15:10:10.393994 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.393958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b202d9b6-1e29-4995-aa00-251fdf0b356b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wrlvf\" (UID: \"b202d9b6-1e29-4995-aa00-251fdf0b356b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" Jul 22 15:10:10.396402 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.396375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b202d9b6-1e29-4995-aa00-251fdf0b356b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wrlvf\" (UID: \"b202d9b6-1e29-4995-aa00-251fdf0b356b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" Jul 22 15:10:10.414870 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.414847 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j5xq\" (UniqueName: \"kubernetes.io/projected/b202d9b6-1e29-4995-aa00-251fdf0b356b-kube-api-access-2j5xq\") pod \"llmisvc-controller-manager-777c6bd7d5-wrlvf\" (UID: \"b202d9b6-1e29-4995-aa00-251fdf0b356b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" Jul 22 15:10:10.491927 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.491873 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" Jul 22 15:10:10.649299 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:10.649276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf"] Jul 22 15:10:10.651491 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:10:10.651458 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb202d9b6_1e29_4995_aa00_251fdf0b356b.slice/crio-6fac80bcec488586f72d7cca47bfe26c470b9094b78654a4ae5f81c8bc1eeca1 WatchSource:0}: Error finding container 6fac80bcec488586f72d7cca47bfe26c470b9094b78654a4ae5f81c8bc1eeca1: Status 404 returned error can't find the container with id 6fac80bcec488586f72d7cca47bfe26c470b9094b78654a4ae5f81c8bc1eeca1 Jul 22 15:10:11.326533 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:11.326491 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" event={"ID":"b202d9b6-1e29-4995-aa00-251fdf0b356b","Type":"ContainerStarted","Data":"6fac80bcec488586f72d7cca47bfe26c470b9094b78654a4ae5f81c8bc1eeca1"} Jul 22 15:10:11.327314 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:11.327291 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nxctp" Jul 22 15:10:14.349176 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:14.349067 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" event={"ID":"b202d9b6-1e29-4995-aa00-251fdf0b356b","Type":"ContainerStarted","Data":"116bcea2099a1ce78abe1357a8cd36f91f2ba5d8dab60b6dbcaf512977ee3218"} Jul 22 15:10:14.349176 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:14.349164 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" Jul 22 15:10:15.293497 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:15.293471 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-ptwkj" Jul 22 15:10:15.296516 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:15.296496 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-bz9tp" Jul 22 15:10:15.314508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:15.314455 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" podStartSLOduration=2.41365844 podStartE2EDuration="5.314436625s" totalImagesPullingTime="2.900778185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:10:14.397163515 +0000 UTC m=+339.896063291" watchObservedRunningTime="2026-07-22 15:10:15.314436625 +0000 UTC m=+340.813336406" Jul 22 15:10:45.356222 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:45.356148 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wrlvf" Jul 22 15:10:59.181700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.181664 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lskvs"] Jul 22 15:10:59.184760 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.184738 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lskvs" Jul 22 15:10:59.187787 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.187767 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 15:10:59.188904 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.188891 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 15:10:59.188974 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.188903 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-nq56m\"" Jul 22 15:10:59.196055 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.196031 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lskvs"] Jul 22 15:10:59.234939 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.234910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vflzb\" (UniqueName: \"kubernetes.io/projected/276437c1-2ea7-4b1b-9909-081fc796f177-kube-api-access-vflzb\") pod \"kuadrant-operator-catalog-lskvs\" (UID: \"276437c1-2ea7-4b1b-9909-081fc796f177\") " pod="kuadrant-system/kuadrant-operator-catalog-lskvs" Jul 22 15:10:59.335945 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.335901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vflzb\" (UniqueName: \"kubernetes.io/projected/276437c1-2ea7-4b1b-9909-081fc796f177-kube-api-access-vflzb\") pod \"kuadrant-operator-catalog-lskvs\" (UID: \"276437c1-2ea7-4b1b-9909-081fc796f177\") " pod="kuadrant-system/kuadrant-operator-catalog-lskvs" Jul 22 15:10:59.344553 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.344526 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vflzb\" (UniqueName: \"kubernetes.io/projected/276437c1-2ea7-4b1b-9909-081fc796f177-kube-api-access-vflzb\") pod \"kuadrant-operator-catalog-lskvs\" (UID: \"276437c1-2ea7-4b1b-9909-081fc796f177\") " pod="kuadrant-system/kuadrant-operator-catalog-lskvs" Jul 22 15:10:59.493734 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.493647 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lskvs" Jul 22 15:10:59.546850 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.546820 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lskvs"] Jul 22 15:10:59.626802 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.626772 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lskvs"] Jul 22 15:10:59.629167 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:10:59.629141 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276437c1_2ea7_4b1b_9909_081fc796f177.slice/crio-3578a6b646fdfad4629ba8644ddbdec0da0d5182520088fbe9b98d4d5547fe35 WatchSource:0}: Error finding container 3578a6b646fdfad4629ba8644ddbdec0da0d5182520088fbe9b98d4d5547fe35: Status 404 returned error can't find the container with id 3578a6b646fdfad4629ba8644ddbdec0da0d5182520088fbe9b98d4d5547fe35 Jul 22 15:10:59.760587 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.760512 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zsrln"] Jul 22 15:10:59.764621 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.764606 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zsrln" Jul 22 15:10:59.774145 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.774110 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zsrln"] Jul 22 15:10:59.839078 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.839052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vqs2\" (UniqueName: \"kubernetes.io/projected/96088db1-f57f-4817-b4b0-d64031c78567-kube-api-access-7vqs2\") pod \"kuadrant-operator-catalog-zsrln\" (UID: \"96088db1-f57f-4817-b4b0-d64031c78567\") " pod="kuadrant-system/kuadrant-operator-catalog-zsrln" Jul 22 15:10:59.940313 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.940284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vqs2\" (UniqueName: \"kubernetes.io/projected/96088db1-f57f-4817-b4b0-d64031c78567-kube-api-access-7vqs2\") pod \"kuadrant-operator-catalog-zsrln\" (UID: \"96088db1-f57f-4817-b4b0-d64031c78567\") " pod="kuadrant-system/kuadrant-operator-catalog-zsrln" Jul 22 15:10:59.949689 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:10:59.949656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vqs2\" (UniqueName: \"kubernetes.io/projected/96088db1-f57f-4817-b4b0-d64031c78567-kube-api-access-7vqs2\") pod \"kuadrant-operator-catalog-zsrln\" (UID: \"96088db1-f57f-4817-b4b0-d64031c78567\") " pod="kuadrant-system/kuadrant-operator-catalog-zsrln" Jul 22 15:11:00.074239 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:00.074212 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zsrln" Jul 22 15:11:00.202006 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:00.201982 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zsrln"] Jul 22 15:11:00.203942 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:11:00.203918 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96088db1_f57f_4817_b4b0_d64031c78567.slice/crio-0eafd3b505ab134ada3c72d1d8ccf06cc1238aeed6bb221cc6a41d947f1e86e9 WatchSource:0}: Error finding container 0eafd3b505ab134ada3c72d1d8ccf06cc1238aeed6bb221cc6a41d947f1e86e9: Status 404 returned error can't find the container with id 0eafd3b505ab134ada3c72d1d8ccf06cc1238aeed6bb221cc6a41d947f1e86e9 Jul 22 15:11:00.530639 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:00.530566 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lskvs" event={"ID":"276437c1-2ea7-4b1b-9909-081fc796f177","Type":"ContainerStarted","Data":"3578a6b646fdfad4629ba8644ddbdec0da0d5182520088fbe9b98d4d5547fe35"} Jul 22 15:11:00.531618 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:00.531590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zsrln" event={"ID":"96088db1-f57f-4817-b4b0-d64031c78567","Type":"ContainerStarted","Data":"0eafd3b505ab134ada3c72d1d8ccf06cc1238aeed6bb221cc6a41d947f1e86e9"} Jul 22 15:11:02.542830 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:02.542724 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zsrln" event={"ID":"96088db1-f57f-4817-b4b0-d64031c78567","Type":"ContainerStarted","Data":"0441fcfa4baf4791cf4b3163ec3239462b5adb7b0b87871d8f232c5fb538f8aa"} Jul 22 15:11:02.544171 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:02.544140 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lskvs" event={"ID":"276437c1-2ea7-4b1b-9909-081fc796f177","Type":"ContainerStarted","Data":"8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63"} Jul 22 15:11:02.544295 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:02.544198 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-lskvs" podUID="276437c1-2ea7-4b1b-9909-081fc796f177" containerName="registry-server" containerID="cri-o://8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63" gracePeriod=2 Jul 22 15:11:02.559175 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:02.559137 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zsrln" podStartSLOduration=1.564312626 podStartE2EDuration="3.559099442s" totalImagesPullingTime="1.994786816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:11:02.558495835 +0000 UTC m=+388.057395611" watchObservedRunningTime="2026-07-22 15:11:02.559099442 +0000 UTC m=+388.057999218" Jul 22 15:11:02.574333 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:02.574298 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lskvs" podStartSLOduration=1.007862971 podStartE2EDuration="3.574285922s" totalImagesPullingTime="2.566422951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:11:02.573501744 +0000 UTC m=+388.072401521" watchObservedRunningTime="2026-07-22 15:11:02.574285922 +0000 UTC m=+388.073185697" Jul 22 15:11:02.787852 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:02.787827 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lskvs" Jul 22 15:11:02.866158 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:02.866132 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vflzb\" (UniqueName: \"kubernetes.io/projected/276437c1-2ea7-4b1b-9909-081fc796f177-kube-api-access-vflzb\") pod \"276437c1-2ea7-4b1b-9909-081fc796f177\" (UID: \"276437c1-2ea7-4b1b-9909-081fc796f177\") " Jul 22 15:11:02.868126 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:02.868093 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276437c1-2ea7-4b1b-9909-081fc796f177-kube-api-access-vflzb" (OuterVolumeSpecName: "kube-api-access-vflzb") pod "276437c1-2ea7-4b1b-9909-081fc796f177" (UID: "276437c1-2ea7-4b1b-9909-081fc796f177"). InnerVolumeSpecName "kube-api-access-vflzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:11:02.967048 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:02.967009 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vflzb\" (UniqueName: \"kubernetes.io/projected/276437c1-2ea7-4b1b-9909-081fc796f177-kube-api-access-vflzb\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:11:03.548910 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:03.548827 2580 generic.go:356] "Generic (PLEG): container finished" podID="276437c1-2ea7-4b1b-9909-081fc796f177" containerID="8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63" exitCode=0 Jul 22 15:11:03.548910 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:03.548871 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lskvs" Jul 22 15:11:03.549361 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:03.548914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lskvs" event={"ID":"276437c1-2ea7-4b1b-9909-081fc796f177","Type":"ContainerDied","Data":"8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63"} Jul 22 15:11:03.549361 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:03.548966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lskvs" event={"ID":"276437c1-2ea7-4b1b-9909-081fc796f177","Type":"ContainerDied","Data":"3578a6b646fdfad4629ba8644ddbdec0da0d5182520088fbe9b98d4d5547fe35"} Jul 22 15:11:03.549361 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:03.548992 2580 scope.go:117] "RemoveContainer" containerID="8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63" Jul 22 15:11:03.557390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:03.557373 2580 scope.go:117] "RemoveContainer" containerID="8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63" Jul 22 15:11:03.557651 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:11:03.557630 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63\": container with ID starting with 8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63 not found: ID does not exist" containerID="8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63" Jul 22 15:11:03.557705 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:03.557660 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63"} err="failed to get container status \"8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63\": rpc error: code = NotFound desc = could not find container \"8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63\": container with ID starting with 8bfcb1d1eb6e089df71dd72b191f9259771dd3c5c672860ea5c567e2708caa63 not found: ID does not exist" Jul 22 15:11:03.580768 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:03.580745 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lskvs"] Jul 22 15:11:03.600310 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:03.600281 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lskvs"] Jul 22 15:11:05.056775 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:05.056744 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276437c1-2ea7-4b1b-9909-081fc796f177" path="/var/lib/kubelet/pods/276437c1-2ea7-4b1b-9909-081fc796f177/volumes" Jul 22 15:11:10.075131 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:10.075093 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-zsrln" Jul 22 15:11:10.075578 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:10.075189 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-zsrln" Jul 22 15:11:10.097309 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:10.097287 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-zsrln" Jul 22 15:11:10.599718 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:10.599690 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-zsrln" Jul 22 15:11:30.731024 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:30.730993 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm"] Jul 22 15:11:30.731497 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:30.731272 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276437c1-2ea7-4b1b-9909-081fc796f177" containerName="registry-server" Jul 22 15:11:30.731497 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:30.731283 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="276437c1-2ea7-4b1b-9909-081fc796f177" containerName="registry-server" Jul 22 15:11:30.731497 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:30.731340 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="276437c1-2ea7-4b1b-9909-081fc796f177" containerName="registry-server" Jul 22 15:11:30.734363 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:30.734347 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm" Jul 22 15:11:30.737230 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:30.737208 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 22 15:11:30.737367 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:30.737275 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q6pbd\"" Jul 22 15:11:30.744903 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:30.744882 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm"] Jul 22 15:11:30.882972 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:30.882945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tczrh\" (UniqueName: \"kubernetes.io/projected/eae0bf69-caf9-4256-bf15-20edc3e97e57-kube-api-access-tczrh\") pod \"dns-operator-controller-manager-864dd78c4c-qvjnm\" (UID: \"eae0bf69-caf9-4256-bf15-20edc3e97e57\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm" Jul 22 15:11:30.984303 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:30.984235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tczrh\" (UniqueName: \"kubernetes.io/projected/eae0bf69-caf9-4256-bf15-20edc3e97e57-kube-api-access-tczrh\") pod \"dns-operator-controller-manager-864dd78c4c-qvjnm\" (UID: \"eae0bf69-caf9-4256-bf15-20edc3e97e57\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm" Jul 22 15:11:31.000709 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:31.000690 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tczrh\" (UniqueName: \"kubernetes.io/projected/eae0bf69-caf9-4256-bf15-20edc3e97e57-kube-api-access-tczrh\") pod \"dns-operator-controller-manager-864dd78c4c-qvjnm\" (UID: \"eae0bf69-caf9-4256-bf15-20edc3e97e57\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm" Jul 22 15:11:31.044551 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:31.044527 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm" Jul 22 15:11:31.174619 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:31.174595 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm"] Jul 22 15:11:31.176923 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:11:31.176898 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae0bf69_caf9_4256_bf15_20edc3e97e57.slice/crio-5c1247ead74042d435ff33c8d2a39a1a8f8a4da94fb021b02a995a2c52edcb8d WatchSource:0}: Error finding container 5c1247ead74042d435ff33c8d2a39a1a8f8a4da94fb021b02a995a2c52edcb8d: Status 404 returned error can't find the container with id 5c1247ead74042d435ff33c8d2a39a1a8f8a4da94fb021b02a995a2c52edcb8d Jul 22 15:11:31.664725 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:31.664696 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm" event={"ID":"eae0bf69-caf9-4256-bf15-20edc3e97e57","Type":"ContainerStarted","Data":"5c1247ead74042d435ff33c8d2a39a1a8f8a4da94fb021b02a995a2c52edcb8d"} Jul 22 15:11:33.675168 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:33.675131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm" event={"ID":"eae0bf69-caf9-4256-bf15-20edc3e97e57","Type":"ContainerStarted","Data":"f8097fbff2cd430530a1471b0726619168fb61975cf34e1d3522ebebff83fe38"} Jul 22 15:11:33.675497 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:33.675213 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm" Jul 22 15:11:33.692786 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:33.692745 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm" podStartSLOduration=1.288725387 podStartE2EDuration="3.69273358s" totalImagesPullingTime="2.404008193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:11:33.69214138 +0000 UTC m=+419.191041159" watchObservedRunningTime="2026-07-22 15:11:33.69273358 +0000 UTC m=+419.191633357" Jul 22 15:11:36.097412 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.097371 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj"] Jul 22 15:11:36.101933 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.101911 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:36.105763 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:11:36.105732 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nb9tj\" is forbidden: User \"system:node:ip-10-0-128-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-22.ec2.internal' and this object" podUID="2e73ecf8-bf4c-4427-8848-8b62b12614ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:36.105892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.105830 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6scrr\"" Jul 22 15:11:36.130041 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.130017 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj"] Jul 22 15:11:36.225740 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.225703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e73ecf8-bf4c-4427-8848-8b62b12614ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nb9tj\" (UID: \"2e73ecf8-bf4c-4427-8848-8b62b12614ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:36.225740 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.225741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5gmh\" (UniqueName: \"kubernetes.io/projected/2e73ecf8-bf4c-4427-8848-8b62b12614ac-kube-api-access-k5gmh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nb9tj\" (UID: \"2e73ecf8-bf4c-4427-8848-8b62b12614ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:36.326876 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.326846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e73ecf8-bf4c-4427-8848-8b62b12614ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nb9tj\" (UID: \"2e73ecf8-bf4c-4427-8848-8b62b12614ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:36.326876 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.326878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gmh\" (UniqueName: \"kubernetes.io/projected/2e73ecf8-bf4c-4427-8848-8b62b12614ac-kube-api-access-k5gmh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nb9tj\" (UID: \"2e73ecf8-bf4c-4427-8848-8b62b12614ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:36.327252 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.327232 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e73ecf8-bf4c-4427-8848-8b62b12614ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nb9tj\" (UID: \"2e73ecf8-bf4c-4427-8848-8b62b12614ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:36.353853 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.353795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5gmh\" (UniqueName: \"kubernetes.io/projected/2e73ecf8-bf4c-4427-8848-8b62b12614ac-kube-api-access-k5gmh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nb9tj\" (UID: \"2e73ecf8-bf4c-4427-8848-8b62b12614ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:36.413282 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.413254 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:36.570778 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.570643 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj"] Jul 22 15:11:36.688976 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:36.688892 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" event={"ID":"2e73ecf8-bf4c-4427-8848-8b62b12614ac","Type":"ContainerStarted","Data":"eba2a92137103725406a89732703083b75151bda6ad6754b0ff219f6fddcb78b"} Jul 22 15:11:42.716614 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:42.716515 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" event={"ID":"2e73ecf8-bf4c-4427-8848-8b62b12614ac","Type":"ContainerStarted","Data":"5266b979bfb0eada31692128e41a0c424a3377cbe5fdfae2d2f377f777ae0114"} Jul 22 15:11:42.717020 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:42.716621 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:42.737558 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:42.737512 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" podStartSLOduration=1.002276125 podStartE2EDuration="6.737499849s" totalImagesPullingTime="5.735223724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:11:42.737232078 +0000 UTC m=+428.236131854" watchObservedRunningTime="2026-07-22 15:11:42.737499849 +0000 UTC m=+428.236399624" Jul 22 15:11:44.681819 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:44.681787 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qvjnm" Jul 22 15:11:53.724032 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:53.723998 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:55.537218 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.537178 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj"] Jul 22 15:11:55.537673 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.537438 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" podUID="2e73ecf8-bf4c-4427-8848-8b62b12614ac" containerName="manager" containerID="cri-o://5266b979bfb0eada31692128e41a0c424a3377cbe5fdfae2d2f377f777ae0114" gracePeriod=2 Jul 22 15:11:55.543220 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.543144 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj"] Jul 22 15:11:55.589182 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.589111 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867"] Jul 22 15:11:55.589406 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.589395 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e73ecf8-bf4c-4427-8848-8b62b12614ac" containerName="manager" Jul 22 15:11:55.589461 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.589408 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e73ecf8-bf4c-4427-8848-8b62b12614ac" containerName="manager" Jul 22 15:11:55.589461 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.589461 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e73ecf8-bf4c-4427-8848-8b62b12614ac" containerName="manager" Jul 22 15:11:55.592207 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.592189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" Jul 22 15:11:55.594720 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:11:55.594695 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nb9tj\" is forbidden: User \"system:node:ip-10-0-128-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-22.ec2.internal' and this object" podUID="2e73ecf8-bf4c-4427-8848-8b62b12614ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:55.616979 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.616957 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867"] Jul 22 15:11:55.751480 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.751448 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt"] Jul 22 15:11:55.754622 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.754603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" Jul 22 15:11:55.757699 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:11:55.757676 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nb9tj\" is forbidden: User \"system:node:ip-10-0-128-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-22.ec2.internal' and this object" podUID="2e73ecf8-bf4c-4427-8848-8b62b12614ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:55.770577 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.770553 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt"] Jul 22 15:11:55.772531 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.772509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/157c3929-7987-40d9-9503-26fecac92432-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9x867\" (UID: \"157c3929-7987-40d9-9503-26fecac92432\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" Jul 22 15:11:55.772627 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.772540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnp9h\" (UniqueName: \"kubernetes.io/projected/157c3929-7987-40d9-9503-26fecac92432-kube-api-access-rnp9h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9x867\" (UID: \"157c3929-7987-40d9-9503-26fecac92432\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" Jul 22 15:11:55.774042 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.774019 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e73ecf8-bf4c-4427-8848-8b62b12614ac" containerID="5266b979bfb0eada31692128e41a0c424a3377cbe5fdfae2d2f377f777ae0114" exitCode=0 Jul 22 15:11:55.774152 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.774088 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eba2a92137103725406a89732703083b75151bda6ad6754b0ff219f6fddcb78b" Jul 22 15:11:55.776914 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.776898 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:55.797442 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:11:55.797389 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nb9tj\" is forbidden: User \"system:node:ip-10-0-128-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-22.ec2.internal' and this object" podUID="2e73ecf8-bf4c-4427-8848-8b62b12614ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:55.873463 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.873437 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e73ecf8-bf4c-4427-8848-8b62b12614ac-extensions-socket-volume\") pod \"2e73ecf8-bf4c-4427-8848-8b62b12614ac\" (UID: \"2e73ecf8-bf4c-4427-8848-8b62b12614ac\") " Jul 22 15:11:55.873576 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.873478 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5gmh\" (UniqueName: \"kubernetes.io/projected/2e73ecf8-bf4c-4427-8848-8b62b12614ac-kube-api-access-k5gmh\") pod \"2e73ecf8-bf4c-4427-8848-8b62b12614ac\" (UID: \"2e73ecf8-bf4c-4427-8848-8b62b12614ac\") " Jul 22 15:11:55.873576 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.873569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/157c3929-7987-40d9-9503-26fecac92432-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9x867\" (UID: \"157c3929-7987-40d9-9503-26fecac92432\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" Jul 22 15:11:55.873691 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.873587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnp9h\" (UniqueName: \"kubernetes.io/projected/157c3929-7987-40d9-9503-26fecac92432-kube-api-access-rnp9h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9x867\" (UID: \"157c3929-7987-40d9-9503-26fecac92432\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" Jul 22 15:11:55.873691 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.873612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25956d58-c707-4674-ab4b-4fe2cb6eaf06-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-25ldt\" (UID: \"25956d58-c707-4674-ab4b-4fe2cb6eaf06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" Jul 22 15:11:55.873691 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.873641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tpd2\" (UniqueName: \"kubernetes.io/projected/25956d58-c707-4674-ab4b-4fe2cb6eaf06-kube-api-access-8tpd2\") pod \"kuadrant-operator-controller-manager-86878c769f-25ldt\" (UID: \"25956d58-c707-4674-ab4b-4fe2cb6eaf06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" Jul 22 15:11:55.873948 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.873927 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/157c3929-7987-40d9-9503-26fecac92432-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9x867\" (UID: \"157c3929-7987-40d9-9503-26fecac92432\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" Jul 22 15:11:55.874007 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.873954 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e73ecf8-bf4c-4427-8848-8b62b12614ac-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2e73ecf8-bf4c-4427-8848-8b62b12614ac" (UID: "2e73ecf8-bf4c-4427-8848-8b62b12614ac"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:11:55.875524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.875505 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e73ecf8-bf4c-4427-8848-8b62b12614ac-kube-api-access-k5gmh" (OuterVolumeSpecName: "kube-api-access-k5gmh") pod "2e73ecf8-bf4c-4427-8848-8b62b12614ac" (UID: "2e73ecf8-bf4c-4427-8848-8b62b12614ac"). InnerVolumeSpecName "kube-api-access-k5gmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:11:55.883050 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.883032 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnp9h\" (UniqueName: \"kubernetes.io/projected/157c3929-7987-40d9-9503-26fecac92432-kube-api-access-rnp9h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9x867\" (UID: \"157c3929-7987-40d9-9503-26fecac92432\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" Jul 22 15:11:55.915035 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.915015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" Jul 22 15:11:55.974379 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.974301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25956d58-c707-4674-ab4b-4fe2cb6eaf06-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-25ldt\" (UID: \"25956d58-c707-4674-ab4b-4fe2cb6eaf06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" Jul 22 15:11:55.974379 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.974359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tpd2\" (UniqueName: \"kubernetes.io/projected/25956d58-c707-4674-ab4b-4fe2cb6eaf06-kube-api-access-8tpd2\") pod \"kuadrant-operator-controller-manager-86878c769f-25ldt\" (UID: \"25956d58-c707-4674-ab4b-4fe2cb6eaf06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" Jul 22 15:11:55.974576 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.974473 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e73ecf8-bf4c-4427-8848-8b62b12614ac-extensions-socket-volume\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:11:55.974576 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.974489 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5gmh\" (UniqueName: \"kubernetes.io/projected/2e73ecf8-bf4c-4427-8848-8b62b12614ac-kube-api-access-k5gmh\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:11:55.974672 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.974608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25956d58-c707-4674-ab4b-4fe2cb6eaf06-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-25ldt\" (UID: \"25956d58-c707-4674-ab4b-4fe2cb6eaf06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" Jul 22 15:11:55.984105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:55.984075 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tpd2\" (UniqueName: \"kubernetes.io/projected/25956d58-c707-4674-ab4b-4fe2cb6eaf06-kube-api-access-8tpd2\") pod \"kuadrant-operator-controller-manager-86878c769f-25ldt\" (UID: \"25956d58-c707-4674-ab4b-4fe2cb6eaf06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" Jul 22 15:11:56.048077 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:56.048054 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867"] Jul 22 15:11:56.049826 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:11:56.049805 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157c3929_7987_40d9_9503_26fecac92432.slice/crio-7e89cc3b3d625eef612ca11e6cb4a3e86c7722b97e525f89b5355e8036d12739 WatchSource:0}: Error finding container 7e89cc3b3d625eef612ca11e6cb4a3e86c7722b97e525f89b5355e8036d12739: Status 404 returned error can't find the container with id 7e89cc3b3d625eef612ca11e6cb4a3e86c7722b97e525f89b5355e8036d12739 Jul 22 15:11:56.072996 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:56.072974 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" Jul 22 15:11:56.206486 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:56.206464 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt"] Jul 22 15:11:56.208725 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:11:56.208695 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25956d58_c707_4674_ab4b_4fe2cb6eaf06.slice/crio-d5b5d21d839167846ebf0bb2e2bd8832072f3cbdf65b02a5ac4dbb6dba5ae986 WatchSource:0}: Error finding container d5b5d21d839167846ebf0bb2e2bd8832072f3cbdf65b02a5ac4dbb6dba5ae986: Status 404 returned error can't find the container with id d5b5d21d839167846ebf0bb2e2bd8832072f3cbdf65b02a5ac4dbb6dba5ae986 Jul 22 15:11:56.779642 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:56.779608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" event={"ID":"25956d58-c707-4674-ab4b-4fe2cb6eaf06","Type":"ContainerStarted","Data":"1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7"} Jul 22 15:11:56.780079 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:56.779699 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" event={"ID":"25956d58-c707-4674-ab4b-4fe2cb6eaf06","Type":"ContainerStarted","Data":"d5b5d21d839167846ebf0bb2e2bd8832072f3cbdf65b02a5ac4dbb6dba5ae986"} Jul 22 15:11:56.780079 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:56.779748 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" Jul 22 15:11:56.781344 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:56.781315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" event={"ID":"157c3929-7987-40d9-9503-26fecac92432","Type":"ContainerStarted","Data":"2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62"} Jul 22 15:11:56.781465 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:56.781363 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" Jul 22 15:11:56.781465 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:56.781385 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" event={"ID":"157c3929-7987-40d9-9503-26fecac92432","Type":"ContainerStarted","Data":"7e89cc3b3d625eef612ca11e6cb4a3e86c7722b97e525f89b5355e8036d12739"} Jul 22 15:11:56.781465 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:56.781387 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:56.782179 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:11:56.782157 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nb9tj\" is forbidden: User \"system:node:ip-10-0-128-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-22.ec2.internal' and this object" podUID="2e73ecf8-bf4c-4427-8848-8b62b12614ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:56.817874 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:56.817825 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" podStartSLOduration=1.817808198 podStartE2EDuration="1.817808198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:11:56.816454137 +0000 UTC m=+442.315353924" watchObservedRunningTime="2026-07-22 15:11:56.817808198 +0000 UTC m=+442.316707977" Jul 22 15:11:56.843647 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:56.843598 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" podStartSLOduration=1.843584286 podStartE2EDuration="1.843584286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:11:56.842248202 +0000 UTC m=+442.341147983" watchObservedRunningTime="2026-07-22 15:11:56.843584286 +0000 UTC m=+442.342484062" Jul 22 15:11:56.845546 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:11:56.845520 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nb9tj\" is forbidden: User \"system:node:ip-10-0-128-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-22.ec2.internal' and this object" podUID="2e73ecf8-bf4c-4427-8848-8b62b12614ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nb9tj" Jul 22 15:11:57.053741 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:11:57.053669 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e73ecf8-bf4c-4427-8848-8b62b12614ac" path="/var/lib/kubelet/pods/2e73ecf8-bf4c-4427-8848-8b62b12614ac/volumes" Jul 22 15:12:07.788140 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:07.788088 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" Jul 22 15:12:07.788613 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:07.788273 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" Jul 22 15:12:07.888096 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:07.888053 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867"] Jul 22 15:12:07.888286 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:07.888258 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" podUID="157c3929-7987-40d9-9503-26fecac92432" containerName="manager" containerID="cri-o://2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62" gracePeriod=10 Jul 22 15:12:08.152128 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.152091 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" Jul 22 15:12:08.166698 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.166678 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnp9h\" (UniqueName: \"kubernetes.io/projected/157c3929-7987-40d9-9503-26fecac92432-kube-api-access-rnp9h\") pod \"157c3929-7987-40d9-9503-26fecac92432\" (UID: \"157c3929-7987-40d9-9503-26fecac92432\") " Jul 22 15:12:08.168557 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.168527 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157c3929-7987-40d9-9503-26fecac92432-kube-api-access-rnp9h" (OuterVolumeSpecName: "kube-api-access-rnp9h") pod "157c3929-7987-40d9-9503-26fecac92432" (UID: "157c3929-7987-40d9-9503-26fecac92432"). InnerVolumeSpecName "kube-api-access-rnp9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:12:08.267503 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.267477 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/157c3929-7987-40d9-9503-26fecac92432-extensions-socket-volume\") pod \"157c3929-7987-40d9-9503-26fecac92432\" (UID: \"157c3929-7987-40d9-9503-26fecac92432\") " Jul 22 15:12:08.267743 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.267726 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnp9h\" (UniqueName: \"kubernetes.io/projected/157c3929-7987-40d9-9503-26fecac92432-kube-api-access-rnp9h\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:12:08.267836 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.267814 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157c3929-7987-40d9-9503-26fecac92432-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "157c3929-7987-40d9-9503-26fecac92432" (UID: "157c3929-7987-40d9-9503-26fecac92432"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:12:08.368742 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.368670 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/157c3929-7987-40d9-9503-26fecac92432-extensions-socket-volume\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:12:08.840002 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.839969 2580 generic.go:356] "Generic (PLEG): container finished" podID="157c3929-7987-40d9-9503-26fecac92432" containerID="2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62" exitCode=0 Jul 22 15:12:08.840392 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.840023 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" event={"ID":"157c3929-7987-40d9-9503-26fecac92432","Type":"ContainerDied","Data":"2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62"} Jul 22 15:12:08.840392 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.840087 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" event={"ID":"157c3929-7987-40d9-9503-26fecac92432","Type":"ContainerDied","Data":"7e89cc3b3d625eef612ca11e6cb4a3e86c7722b97e525f89b5355e8036d12739"} Jul 22 15:12:08.840392 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.840028 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867" Jul 22 15:12:08.840392 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.840132 2580 scope.go:117] "RemoveContainer" containerID="2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62" Jul 22 15:12:08.848998 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.848979 2580 scope.go:117] "RemoveContainer" containerID="2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62" Jul 22 15:12:08.849271 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:12:08.849249 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62\": container with ID starting with 2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62 not found: ID does not exist" containerID="2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62" Jul 22 15:12:08.849369 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.849276 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62"} err="failed to get container status \"2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62\": rpc error: code = NotFound desc = could not find container \"2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62\": container with ID starting with 2dedb6670698cedf50c50364d3603d21bae006ca59ecbbdc81c7a1e252324a62 not found: ID does not exist" Jul 22 15:12:08.883609 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.883586 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867"] Jul 22 15:12:08.894171 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:08.894149 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9x867"] Jul 22 15:12:09.053737 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:09.053708 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="157c3929-7987-40d9-9503-26fecac92432" path="/var/lib/kubelet/pods/157c3929-7987-40d9-9503-26fecac92432/volumes" Jul 22 15:12:24.288216 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.287674 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9"] Jul 22 15:12:24.288660 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.288478 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="157c3929-7987-40d9-9503-26fecac92432" containerName="manager" Jul 22 15:12:24.288660 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.288499 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="157c3929-7987-40d9-9503-26fecac92432" containerName="manager" Jul 22 15:12:24.288741 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.288697 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="157c3929-7987-40d9-9503-26fecac92432" containerName="manager" Jul 22 15:12:24.291447 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.291428 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.295072 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.295053 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jrbmx\"" Jul 22 15:12:24.301486 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.301465 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9"] Jul 22 15:12:24.379147 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.379088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.379402 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.379373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxhk\" (UniqueName: \"kubernetes.io/projected/f32c102b-7deb-4be1-a86b-a060709ae9b0-kube-api-access-6pxhk\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.379546 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.379423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.379546 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.379501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32c102b-7deb-4be1-a86b-a060709ae9b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.379658 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.379552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.379658 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.379594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32c102b-7deb-4be1-a86b-a060709ae9b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.379658 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.379619 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.379658 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.379637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.379793 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.379685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32c102b-7deb-4be1-a86b-a060709ae9b0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481075 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32c102b-7deb-4be1-a86b-a060709ae9b0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481292 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481292 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481255 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxhk\" (UniqueName: \"kubernetes.io/projected/f32c102b-7deb-4be1-a86b-a060709ae9b0-kube-api-access-6pxhk\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481422 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481422 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32c102b-7deb-4be1-a86b-a060709ae9b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481422 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481381 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481570 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32c102b-7deb-4be1-a86b-a060709ae9b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481570 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481460 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481570 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481481 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481570 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481778 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481836 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.481955 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.481933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.482053 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.482007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32c102b-7deb-4be1-a86b-a060709ae9b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.483705 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.483680 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32c102b-7deb-4be1-a86b-a060709ae9b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.483811 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.483792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32c102b-7deb-4be1-a86b-a060709ae9b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.489183 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.489165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32c102b-7deb-4be1-a86b-a060709ae9b0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.489321 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.489305 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxhk\" (UniqueName: \"kubernetes.io/projected/f32c102b-7deb-4be1-a86b-a060709ae9b0-kube-api-access-6pxhk\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl7q9\" (UID: \"f32c102b-7deb-4be1-a86b-a060709ae9b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.605071 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.605045 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:24.739636 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.739487 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9"] Jul 22 15:12:24.741637 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:12:24.741603 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32c102b_7deb_4be1_a86b_a060709ae9b0.slice/crio-210d60d9583340d1c43f50a07ad383dcd495fc3b552cbd2cf87525eb2733d0b6 WatchSource:0}: Error finding container 210d60d9583340d1c43f50a07ad383dcd495fc3b552cbd2cf87525eb2733d0b6: Status 404 returned error can't find the container with id 210d60d9583340d1c43f50a07ad383dcd495fc3b552cbd2cf87525eb2733d0b6 Jul 22 15:12:24.911486 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:24.911421 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" event={"ID":"f32c102b-7deb-4be1-a86b-a060709ae9b0","Type":"ContainerStarted","Data":"210d60d9583340d1c43f50a07ad383dcd495fc3b552cbd2cf87525eb2733d0b6"} Jul 22 15:12:27.207490 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:27.207451 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:12:27.207905 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:27.207528 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:12:27.207905 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:27.207558 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:12:27.928180 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:27.928147 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" event={"ID":"f32c102b-7deb-4be1-a86b-a060709ae9b0","Type":"ContainerStarted","Data":"7ca7e1895377dc5be46c49f07451ea2e8cfc0e09bd02e78433c1f459229b6988"} Jul 22 15:12:27.950549 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:27.950492 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" podStartSLOduration=1.486962826 podStartE2EDuration="3.950473316s" totalImagesPullingTime="2.46351049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:12:27.94987358 +0000 UTC m=+473.448773357" watchObservedRunningTime="2026-07-22 15:12:27.950473316 +0000 UTC m=+473.449373094" Jul 22 15:12:28.605568 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:28.605532 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:28.609955 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:28.609932 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:28.932596 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:28.932517 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:28.933395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:28.933374 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:12:29.503326 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.503289 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:12:29.505693 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.505676 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" Jul 22 15:12:29.508845 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.508822 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4gzh6\"" Jul 22 15:12:29.508979 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.508960 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 15:12:29.515688 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.515667 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:12:29.525685 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.525666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/00778b00-ec3b-4341-8166-6474593daafb-config-file\") pod \"limitador-limitador-54d6d988d5-d697m\" (UID: \"00778b00-ec3b-4341-8166-6474593daafb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" Jul 22 15:12:29.525780 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.525705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59g5\" (UniqueName: \"kubernetes.io/projected/00778b00-ec3b-4341-8166-6474593daafb-kube-api-access-n59g5\") pod \"limitador-limitador-54d6d988d5-d697m\" (UID: \"00778b00-ec3b-4341-8166-6474593daafb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" Jul 22 15:12:29.554029 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.554005 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:12:29.626326 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.626303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59g5\" (UniqueName: \"kubernetes.io/projected/00778b00-ec3b-4341-8166-6474593daafb-kube-api-access-n59g5\") pod \"limitador-limitador-54d6d988d5-d697m\" (UID: \"00778b00-ec3b-4341-8166-6474593daafb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" Jul 22 15:12:29.626625 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.626398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/00778b00-ec3b-4341-8166-6474593daafb-config-file\") pod \"limitador-limitador-54d6d988d5-d697m\" (UID: \"00778b00-ec3b-4341-8166-6474593daafb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" Jul 22 15:12:29.626965 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.626948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/00778b00-ec3b-4341-8166-6474593daafb-config-file\") pod \"limitador-limitador-54d6d988d5-d697m\" (UID: \"00778b00-ec3b-4341-8166-6474593daafb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" Jul 22 15:12:29.635224 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.635200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59g5\" (UniqueName: \"kubernetes.io/projected/00778b00-ec3b-4341-8166-6474593daafb-kube-api-access-n59g5\") pod \"limitador-limitador-54d6d988d5-d697m\" (UID: \"00778b00-ec3b-4341-8166-6474593daafb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" Jul 22 15:12:29.816421 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.816394 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" Jul 22 15:12:29.949077 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:29.949049 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:12:29.950041 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:12:29.950018 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00778b00_ec3b_4341_8166_6474593daafb.slice/crio-1d1ea7eb6dd1101da09afeb1d18d1704587416dd77f210e8d3cd8a2d3ad9725d WatchSource:0}: Error finding container 1d1ea7eb6dd1101da09afeb1d18d1704587416dd77f210e8d3cd8a2d3ad9725d: Status 404 returned error can't find the container with id 1d1ea7eb6dd1101da09afeb1d18d1704587416dd77f210e8d3cd8a2d3ad9725d Jul 22 15:12:30.945210 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:30.945178 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" event={"ID":"00778b00-ec3b-4341-8166-6474593daafb","Type":"ContainerStarted","Data":"1d1ea7eb6dd1101da09afeb1d18d1704587416dd77f210e8d3cd8a2d3ad9725d"} Jul 22 15:12:32.954533 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:32.954440 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" event={"ID":"00778b00-ec3b-4341-8166-6474593daafb","Type":"ContainerStarted","Data":"b6c1e7ed27a519d168b6c677b954117c88d097e609bb9764f7cfbe27e8a57b80"} Jul 22 15:12:32.954533 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:32.954526 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" Jul 22 15:12:32.972465 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:32.972425 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" podStartSLOduration=1.289020054 podStartE2EDuration="3.972411066s" totalImagesPullingTime="2.683391012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:12:32.971448014 +0000 UTC m=+478.470347781" watchObservedRunningTime="2026-07-22 15:12:32.972411066 +0000 UTC m=+478.471310886" Jul 22 15:12:43.959738 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:12:43.959708 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-d697m" Jul 22 15:13:41.788215 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:41.788137 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:13:50.278368 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:50.278337 2580 generic.go:356] "Generic (PLEG): container finished" podID="f32c102b-7deb-4be1-a86b-a060709ae9b0" containerID="7ca7e1895377dc5be46c49f07451ea2e8cfc0e09bd02e78433c1f459229b6988" exitCode=0 Jul 22 15:13:50.278885 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:50.278415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" event={"ID":"f32c102b-7deb-4be1-a86b-a060709ae9b0","Type":"ContainerDied","Data":"7ca7e1895377dc5be46c49f07451ea2e8cfc0e09bd02e78433c1f459229b6988"} Jul 22 15:13:50.278885 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:50.278867 2580 scope.go:117] "RemoveContainer" containerID="7ca7e1895377dc5be46c49f07451ea2e8cfc0e09bd02e78433c1f459229b6988" Jul 22 15:13:50.279950 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:50.279923 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:13:50.280010 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:50.279989 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:13:50.280044 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:50.280019 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:13:51.284159 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:51.284103 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" event={"ID":"f32c102b-7deb-4be1-a86b-a060709ae9b0","Type":"ContainerStarted","Data":"5a85134e70831b89bdd8db0512858cac87b57f7ecdb1d2410108d984569c5e99"} Jul 22 15:13:51.605588 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:51.605546 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:13:51.605829 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:51.605805 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cl7q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Jul 22 15:13:51.605905 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:51.605887 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" podUID="f32c102b-7deb-4be1-a86b-a060709ae9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Jul 22 15:13:52.605507 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:52.605467 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cl7q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Jul 22 15:13:52.605945 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:52.605525 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" podUID="f32c102b-7deb-4be1-a86b-a060709ae9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Jul 22 15:13:53.605990 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:53.605955 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cl7q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Jul 22 15:13:53.606373 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:53.606037 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" podUID="f32c102b-7deb-4be1-a86b-a060709ae9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Jul 22 15:13:54.605992 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:54.605953 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cl7q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Jul 22 15:13:54.606205 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:54.606011 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" podUID="f32c102b-7deb-4be1-a86b-a060709ae9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Jul 22 15:13:55.609262 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:55.609229 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:13:55.609725 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:55.609621 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:13:55.610316 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:13:55.610295 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl7q9" Jul 22 15:14:00.179895 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:00.179862 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:14:05.668369 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.668332 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-snf67"] Jul 22 15:14:05.670516 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.670501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.675317 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.675298 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 15:14:05.676722 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.676692 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 15:14:05.676722 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.676717 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9hwbl\"" Jul 22 15:14:05.677190 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.676938 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 22 15:14:05.690521 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.690501 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-snf67"] Jul 22 15:14:05.790966 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.790937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.791113 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.790985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.791113 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.791047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.791237 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.791152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.791237 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.791187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbrb\" (UniqueName: \"kubernetes.io/projected/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-kube-api-access-zsbrb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.791237 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.791219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.892373 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.892343 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.892373 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.892375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.892564 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.892505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.892605 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.892576 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.892648 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.892603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbrb\" (UniqueName: \"kubernetes.io/projected/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-kube-api-access-zsbrb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.892648 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.892630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.892778 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.892751 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.892905 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.892875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.893025 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.892918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.894706 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.894685 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.895092 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.895072 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.902112 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.902093 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbrb\" (UniqueName: \"kubernetes.io/projected/d63a3b0f-12e3-454e-9348-57c7c8cff3a5-kube-api-access-zsbrb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-snf67\" (UID: \"d63a3b0f-12e3-454e-9348-57c7c8cff3a5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:05.981147 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:05.981080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:06.109950 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:06.109928 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-snf67"] Jul 22 15:14:06.112502 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:14:06.112462 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63a3b0f_12e3_454e_9348_57c7c8cff3a5.slice/crio-03c85d7bcf55eb8c632d60c2a0065ab98dc0819bd4846c955856cb1e2e6e9736 WatchSource:0}: Error finding container 03c85d7bcf55eb8c632d60c2a0065ab98dc0819bd4846c955856cb1e2e6e9736: Status 404 returned error can't find the container with id 03c85d7bcf55eb8c632d60c2a0065ab98dc0819bd4846c955856cb1e2e6e9736 Jul 22 15:14:06.348719 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:06.348680 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" event={"ID":"d63a3b0f-12e3-454e-9348-57c7c8cff3a5","Type":"ContainerStarted","Data":"03c85d7bcf55eb8c632d60c2a0065ab98dc0819bd4846c955856cb1e2e6e9736"} Jul 22 15:14:06.579186 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:06.579156 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:14:08.359087 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:08.359051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" event={"ID":"d63a3b0f-12e3-454e-9348-57c7c8cff3a5","Type":"ContainerStarted","Data":"f3526d7eb713dd235f7d9a0de6e5dd7200211955a55191911978567bb7ddb6e0"} Jul 22 15:14:08.380873 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:08.380829 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" podStartSLOduration=2.024033571 podStartE2EDuration="3.380814347s" totalImagesPullingTime="1.356780776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:14:08.37895719 +0000 UTC m=+573.877856968" watchObservedRunningTime="2026-07-22 15:14:08.380814347 +0000 UTC m=+573.879714153" Jul 22 15:14:15.981525 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:15.981492 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:15.981942 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:15.981569 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:15.993909 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:15.993879 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:16.404598 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:16.404570 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-snf67" Jul 22 15:14:20.679050 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:20.679015 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:14:31.071936 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:31.071903 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:14:38.121805 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:38.121770 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:14:48.980602 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:14:48.980564 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:15:00.167731 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:00.167659 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745555-nrwd7"] Jul 22 15:15:00.170036 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:00.170020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745555-nrwd7" Jul 22 15:15:00.173306 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:00.173285 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b79tk\"" Jul 22 15:15:00.179511 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:00.179493 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745555-nrwd7" Jul 22 15:15:00.191718 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:00.191695 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745555-nrwd7"] Jul 22 15:15:00.310143 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:00.310072 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745555-nrwd7"] Jul 22 15:15:00.312327 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:15:00.312300 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f016b3_8e40_4cbe_a4f4_533a68c9bec3.slice/crio-0116b2ef63a25147bb975bdf16fb0367f135ac62e2e71e506c8b926cc0ea7c56 WatchSource:0}: Error finding container 0116b2ef63a25147bb975bdf16fb0367f135ac62e2e71e506c8b926cc0ea7c56: Status 404 returned error can't find the container with id 0116b2ef63a25147bb975bdf16fb0367f135ac62e2e71e506c8b926cc0ea7c56 Jul 22 15:15:00.314348 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:00.314329 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 15:15:00.589178 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:00.589148 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745555-nrwd7" event={"ID":"c7f016b3-8e40-4cbe-a4f4-533a68c9bec3","Type":"ContainerStarted","Data":"0116b2ef63a25147bb975bdf16fb0367f135ac62e2e71e506c8b926cc0ea7c56"} Jul 22 15:15:03.604389 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:03.604350 2580 generic.go:356] "Generic (PLEG): container finished" podID="c7f016b3-8e40-4cbe-a4f4-533a68c9bec3" containerID="fe836298b6c041a17451d40571ff142ca9ede14e5f1043959584cc13bea7e155" exitCode=0 Jul 22 15:15:03.604741 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:03.604401 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745555-nrwd7" event={"ID":"c7f016b3-8e40-4cbe-a4f4-533a68c9bec3","Type":"ContainerDied","Data":"fe836298b6c041a17451d40571ff142ca9ede14e5f1043959584cc13bea7e155"} Jul 22 15:15:04.731082 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:04.731056 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745555-nrwd7" Jul 22 15:15:05.614171 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:05.614144 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745555-nrwd7" Jul 22 15:15:05.614333 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:05.614144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745555-nrwd7" event={"ID":"c7f016b3-8e40-4cbe-a4f4-533a68c9bec3","Type":"ContainerDied","Data":"0116b2ef63a25147bb975bdf16fb0367f135ac62e2e71e506c8b926cc0ea7c56"} Jul 22 15:15:05.614333 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:05.614262 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0116b2ef63a25147bb975bdf16fb0367f135ac62e2e71e506c8b926cc0ea7c56" Jul 22 15:15:43.584365 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:43.584327 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:15:53.278681 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:15:53.278651 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:16:03.376313 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:16:03.376282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:17:46.685161 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:17:46.685125 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:17:58.482595 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:17:58.482517 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:18:07.179714 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:18:07.179678 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:18:19.089135 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:18:19.089086 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:18:34.953419 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:18:34.953387 2580 scope.go:117] "RemoveContainer" containerID="5266b979bfb0eada31692128e41a0c424a3377cbe5fdfae2d2f377f777ae0114" Jul 22 15:18:38.983355 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:18:38.983321 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:18:46.492055 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:18:46.492027 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:18:55.817240 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:18:55.817199 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:19:02.374390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:19:02.374352 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:19:12.281220 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:19:12.281186 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:19:21.383804 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:19:21.383767 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:19:32.284359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:19:32.284273 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:19:42.287157 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:19:42.287126 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:20:04.572966 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:20:04.572933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:20:16.195707 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:20:16.195678 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:20:41.271698 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:20:41.271661 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:20:54.869144 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:20:54.869100 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:21:05.674525 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:21:05.674441 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:21:15.276346 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:21:15.276312 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:21:24.684602 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:21:24.684550 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:21:34.676139 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:21:34.676086 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:21:44.076057 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:21:44.076019 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:22:10.621361 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:10.621324 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt"] Jul 22 15:22:10.621901 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:10.621549 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" podUID="25956d58-c707-4674-ab4b-4fe2cb6eaf06" containerName="manager" containerID="cri-o://1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7" gracePeriod=10 Jul 22 15:22:13.274201 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.274177 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" Jul 22 15:22:13.314038 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.314014 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25956d58-c707-4674-ab4b-4fe2cb6eaf06-extensions-socket-volume\") pod \"25956d58-c707-4674-ab4b-4fe2cb6eaf06\" (UID: \"25956d58-c707-4674-ab4b-4fe2cb6eaf06\") " Jul 22 15:22:13.314218 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.314067 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tpd2\" (UniqueName: \"kubernetes.io/projected/25956d58-c707-4674-ab4b-4fe2cb6eaf06-kube-api-access-8tpd2\") pod \"25956d58-c707-4674-ab4b-4fe2cb6eaf06\" (UID: \"25956d58-c707-4674-ab4b-4fe2cb6eaf06\") " Jul 22 15:22:13.314493 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.314468 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25956d58-c707-4674-ab4b-4fe2cb6eaf06-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "25956d58-c707-4674-ab4b-4fe2cb6eaf06" (UID: "25956d58-c707-4674-ab4b-4fe2cb6eaf06"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:22:13.316094 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.316069 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25956d58-c707-4674-ab4b-4fe2cb6eaf06-kube-api-access-8tpd2" (OuterVolumeSpecName: "kube-api-access-8tpd2") pod "25956d58-c707-4674-ab4b-4fe2cb6eaf06" (UID: "25956d58-c707-4674-ab4b-4fe2cb6eaf06"). InnerVolumeSpecName "kube-api-access-8tpd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:22:13.415647 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.415564 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tpd2\" (UniqueName: \"kubernetes.io/projected/25956d58-c707-4674-ab4b-4fe2cb6eaf06-kube-api-access-8tpd2\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:22:13.415647 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.415591 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25956d58-c707-4674-ab4b-4fe2cb6eaf06-extensions-socket-volume\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:22:13.494629 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.494597 2580 generic.go:356] "Generic (PLEG): container finished" podID="25956d58-c707-4674-ab4b-4fe2cb6eaf06" containerID="1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7" exitCode=0 Jul 22 15:22:13.494793 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.494638 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" Jul 22 15:22:13.494793 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.494648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" event={"ID":"25956d58-c707-4674-ab4b-4fe2cb6eaf06","Type":"ContainerDied","Data":"1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7"} Jul 22 15:22:13.494793 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.494693 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt" event={"ID":"25956d58-c707-4674-ab4b-4fe2cb6eaf06","Type":"ContainerDied","Data":"d5b5d21d839167846ebf0bb2e2bd8832072f3cbdf65b02a5ac4dbb6dba5ae986"} Jul 22 15:22:13.494793 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.494722 2580 scope.go:117] "RemoveContainer" containerID="1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7" Jul 22 15:22:13.506404 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.506387 2580 scope.go:117] "RemoveContainer" containerID="1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7" Jul 22 15:22:13.506656 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:22:13.506635 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7\": container with ID starting with 1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7 not found: ID does not exist" containerID="1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7" Jul 22 15:22:13.506722 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.506665 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7"} err="failed to get container status \"1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7\": rpc error: code = NotFound desc = could not find container \"1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7\": container with ID starting with 1811356fe2ce8d6405ae2cf9a0e2833c52e2076456d22956afb20d558c4481a7 not found: ID does not exist" Jul 22 15:22:13.521095 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.521066 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt"] Jul 22 15:22:13.526104 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:13.526085 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-25ldt"] Jul 22 15:22:15.054928 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:22:15.054893 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25956d58-c707-4674-ab4b-4fe2cb6eaf06" path="/var/lib/kubelet/pods/25956d58-c707-4674-ab4b-4fe2cb6eaf06/volumes" Jul 22 15:23:11.824824 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.824784 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq"] Jul 22 15:23:11.825489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.825318 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25956d58-c707-4674-ab4b-4fe2cb6eaf06" containerName="manager" Jul 22 15:23:11.825489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.825339 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="25956d58-c707-4674-ab4b-4fe2cb6eaf06" containerName="manager" Jul 22 15:23:11.825489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.825353 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f016b3-8e40-4cbe-a4f4-533a68c9bec3" containerName="cleanup" Jul 22 15:23:11.825489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.825362 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f016b3-8e40-4cbe-a4f4-533a68c9bec3" containerName="cleanup" Jul 22 15:23:11.825489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.825439 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f016b3-8e40-4cbe-a4f4-533a68c9bec3" containerName="cleanup" Jul 22 15:23:11.825489 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.825451 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="25956d58-c707-4674-ab4b-4fe2cb6eaf06" containerName="manager" Jul 22 15:23:11.828311 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.828290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" Jul 22 15:23:11.832032 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.832010 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6scrr\"" Jul 22 15:23:11.847847 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.847821 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq"] Jul 22 15:23:11.883550 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.883519 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d27f993d-22bb-4636-ae76-cc7981d2cd72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hp5rq\" (UID: \"d27f993d-22bb-4636-ae76-cc7981d2cd72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" Jul 22 15:23:11.883550 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.883552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s69zz\" (UniqueName: \"kubernetes.io/projected/d27f993d-22bb-4636-ae76-cc7981d2cd72-kube-api-access-s69zz\") pod \"kuadrant-operator-controller-manager-86878c769f-hp5rq\" (UID: \"d27f993d-22bb-4636-ae76-cc7981d2cd72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" Jul 22 15:23:11.984543 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.984506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d27f993d-22bb-4636-ae76-cc7981d2cd72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hp5rq\" (UID: \"d27f993d-22bb-4636-ae76-cc7981d2cd72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" Jul 22 15:23:11.984543 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.984542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s69zz\" (UniqueName: \"kubernetes.io/projected/d27f993d-22bb-4636-ae76-cc7981d2cd72-kube-api-access-s69zz\") pod \"kuadrant-operator-controller-manager-86878c769f-hp5rq\" (UID: \"d27f993d-22bb-4636-ae76-cc7981d2cd72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" Jul 22 15:23:11.984900 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:11.984879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d27f993d-22bb-4636-ae76-cc7981d2cd72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hp5rq\" (UID: \"d27f993d-22bb-4636-ae76-cc7981d2cd72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" Jul 22 15:23:12.004535 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:12.004506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s69zz\" (UniqueName: \"kubernetes.io/projected/d27f993d-22bb-4636-ae76-cc7981d2cd72-kube-api-access-s69zz\") pod \"kuadrant-operator-controller-manager-86878c769f-hp5rq\" (UID: \"d27f993d-22bb-4636-ae76-cc7981d2cd72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" Jul 22 15:23:12.138553 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:12.138468 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" Jul 22 15:23:12.288014 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:12.287972 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq"] Jul 22 15:23:12.292151 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:23:12.292112 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27f993d_22bb_4636_ae76_cc7981d2cd72.slice/crio-61f4ed33097cdddad66658a341ef3dbff7ab87e0f438c467433a36d0abd1bbcf WatchSource:0}: Error finding container 61f4ed33097cdddad66658a341ef3dbff7ab87e0f438c467433a36d0abd1bbcf: Status 404 returned error can't find the container with id 61f4ed33097cdddad66658a341ef3dbff7ab87e0f438c467433a36d0abd1bbcf Jul 22 15:23:12.746893 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:12.746852 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" event={"ID":"d27f993d-22bb-4636-ae76-cc7981d2cd72","Type":"ContainerStarted","Data":"b86d88e4525931785f90a1c1f723763ff32249449b36cb5768d7aee3c2b5fbba"} Jul 22 15:23:12.747064 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:12.746913 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" event={"ID":"d27f993d-22bb-4636-ae76-cc7981d2cd72","Type":"ContainerStarted","Data":"61f4ed33097cdddad66658a341ef3dbff7ab87e0f438c467433a36d0abd1bbcf"} Jul 22 15:23:12.747064 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:12.746950 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" Jul 22 15:23:12.835218 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:12.835149 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" podStartSLOduration=1.8351315879999999 podStartE2EDuration="1.835131588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:23:12.829537792 +0000 UTC m=+1118.328437664" watchObservedRunningTime="2026-07-22 15:23:12.835131588 +0000 UTC m=+1118.334031354" Jul 22 15:23:23.753542 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:23.753513 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5rq" Jul 22 15:23:33.072561 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:33.072523 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:23:42.576631 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:42.576595 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:23:55.270862 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:23:55.270829 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:24:07.577249 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:24:07.577170 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:24:17.165487 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:24:17.165454 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:24:28.780215 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:24:28.780177 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:24:37.470909 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:24:37.470874 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:24:48.072429 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:24:48.072393 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:24:59.871346 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:24:59.871308 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:25:08.463697 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:25:08.463662 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:25:19.874567 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:25:19.874531 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:25:41.476350 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:25:41.476272 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:25:51.478189 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:25:51.478158 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:26:09.475331 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:26:09.475297 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:26:25.614293 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:26:25.614262 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:26:36.369037 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:26:36.369006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:26:43.163481 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:26:43.163448 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:26:57.572770 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:26:57.572695 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:27:08.072475 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:27:08.072438 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:27:17.465973 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:27:17.465937 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:27:27.874153 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:27:27.874107 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:27:37.469311 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:27:37.469275 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:27:46.776652 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:27:46.776622 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:27:59.568760 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:27:59.568727 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:28:08.471257 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:28:08.471223 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:28:18.372915 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:28:18.372881 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:28:28.976448 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:28:28.976371 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:28:38.569665 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:28:38.569631 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:28:45.472297 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:28:45.472266 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:28:55.273627 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:28:55.273596 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:29:03.372211 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:03.372173 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:29:15.375937 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:15.375905 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:29:24.665180 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:24.665143 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:29:39.082863 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:39.082828 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:29:48.167550 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:48.167510 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:29:57.570191 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:57.570107 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:29:59.690245 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.690213 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g"] Jul 22 15:29:59.693447 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.693429 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.697178 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.697151 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c6b1b943-openshift-default-dockercfg-5xdrw\"" Jul 22 15:29:59.735839 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.735812 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g"] Jul 22 15:29:59.814659 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.814623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv9fn\" (UniqueName: \"kubernetes.io/projected/658ec699-801c-4f6d-b886-56984effd723-kube-api-access-xv9fn\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.814824 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.814704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-workload-socket\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.814824 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.814731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-credential-socket\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.814824 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.814756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-istio-envoy\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.814929 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.814876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658ec699-801c-4f6d-b886-56984effd723-istio-token\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.814980 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.814936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-istio-data\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.815017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.814983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658ec699-801c-4f6d-b886-56984effd723-istiod-ca-cert\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.815050 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.815031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-workload-certs\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.815083 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.815057 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658ec699-801c-4f6d-b886-56984effd723-istio-podinfo\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.916490 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.916455 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-workload-certs\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.916659 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.916497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658ec699-801c-4f6d-b886-56984effd723-istio-podinfo\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.916659 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.916546 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv9fn\" (UniqueName: \"kubernetes.io/projected/658ec699-801c-4f6d-b886-56984effd723-kube-api-access-xv9fn\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.916788 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.916766 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-workload-socket\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.916842 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.916807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-credential-socket\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.916842 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.916819 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-workload-certs\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.916941 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.916840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-istio-envoy\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.916941 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.916877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658ec699-801c-4f6d-b886-56984effd723-istio-token\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.916941 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.916926 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-istio-data\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.917080 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.916956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658ec699-801c-4f6d-b886-56984effd723-istiod-ca-cert\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.917080 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.917045 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-workload-socket\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.917268 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.917170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-credential-socket\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.917319 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.917302 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-istio-data\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.917666 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.917646 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658ec699-801c-4f6d-b886-56984effd723-istiod-ca-cert\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.919292 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.919271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-istio-envoy\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.919745 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.919725 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658ec699-801c-4f6d-b886-56984effd723-istio-podinfo\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.925758 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.925731 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv9fn\" (UniqueName: \"kubernetes.io/projected/658ec699-801c-4f6d-b886-56984effd723-kube-api-access-xv9fn\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:29:59.925992 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:29:59.925974 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658ec699-801c-4f6d-b886-56984effd723-istio-token\") pod \"e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:30:00.004235 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:00.004160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:30:00.151071 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:00.151036 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g"] Jul 22 15:30:00.154383 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:30:00.154347 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658ec699_801c_4f6d_b886_56984effd723.slice/crio-00f42c600d98a2fc029d205a190aff71969400a21b172afffaef52e55e8ba294 WatchSource:0}: Error finding container 00f42c600d98a2fc029d205a190aff71969400a21b172afffaef52e55e8ba294: Status 404 returned error can't find the container with id 00f42c600d98a2fc029d205a190aff71969400a21b172afffaef52e55e8ba294 Jul 22 15:30:00.157387 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:00.157351 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:30:00.157502 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:00.157431 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:30:00.157502 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:00.157480 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:30:00.506038 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:00.506003 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" event={"ID":"658ec699-801c-4f6d-b886-56984effd723","Type":"ContainerStarted","Data":"58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7"} Jul 22 15:30:00.506223 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:00.506061 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" event={"ID":"658ec699-801c-4f6d-b886-56984effd723","Type":"ContainerStarted","Data":"00f42c600d98a2fc029d205a190aff71969400a21b172afffaef52e55e8ba294"} Jul 22 15:30:00.531264 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:00.531219 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" podStartSLOduration=1.531203292 podStartE2EDuration="1.531203292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:30:00.528579413 +0000 UTC m=+1526.027479204" watchObservedRunningTime="2026-07-22 15:30:00.531203292 +0000 UTC m=+1526.030103068" Jul 22 15:30:01.004766 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:01.004728 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:30:01.006130 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:01.006092 2580 patch_prober.go:28] interesting pod/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 22 15:30:01.006209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:01.006159 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 22 15:30:02.009643 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.009607 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:30:02.094658 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.094621 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq"] Jul 22 15:30:02.098301 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.098278 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:02.101513 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.101488 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 22 15:30:02.101639 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.101494 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 15:30:02.101639 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.101590 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8q48q\"" Jul 22 15:30:02.101793 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.101777 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c6b1b943-serving-cert\"" Jul 22 15:30:02.112856 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.112836 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq"] Jul 22 15:30:02.134070 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.134050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b6c0d-46cc-4fb8-97df-91323202eef9-maas-api-tls\") pod \"maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:02.134223 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.134078 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b6c0d-46cc-4fb8-97df-91323202eef9-openshift-service-ca\") pod \"maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:02.134223 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.134103 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm598\" (UniqueName: \"kubernetes.io/projected/533b6c0d-46cc-4fb8-97df-91323202eef9-kube-api-access-pm598\") pod \"maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:02.234782 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.234750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm598\" (UniqueName: \"kubernetes.io/projected/533b6c0d-46cc-4fb8-97df-91323202eef9-kube-api-access-pm598\") pod \"maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:02.234933 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.234869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b6c0d-46cc-4fb8-97df-91323202eef9-maas-api-tls\") pod \"maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:02.234933 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.234896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b6c0d-46cc-4fb8-97df-91323202eef9-openshift-service-ca\") pod \"maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:02.235029 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:30:02.235009 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-serving-cert: secret "maas-api-e2e-ait-c6b1b943-serving-cert" not found Jul 22 15:30:02.235081 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:30:02.235070 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533b6c0d-46cc-4fb8-97df-91323202eef9-maas-api-tls podName:533b6c0d-46cc-4fb8-97df-91323202eef9 nodeName:}" failed. No retries permitted until 2026-07-22 15:30:02.735054414 +0000 UTC m=+1528.233954171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/533b6c0d-46cc-4fb8-97df-91323202eef9-maas-api-tls") pod "maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" (UID: "533b6c0d-46cc-4fb8-97df-91323202eef9") : secret "maas-api-e2e-ait-c6b1b943-serving-cert" not found Jul 22 15:30:02.235716 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.235697 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b6c0d-46cc-4fb8-97df-91323202eef9-openshift-service-ca\") pod \"maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:02.244724 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.244704 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm598\" (UniqueName: \"kubernetes.io/projected/533b6c0d-46cc-4fb8-97df-91323202eef9-kube-api-access-pm598\") pod \"maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:02.516837 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.516803 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:30:02.517811 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.517790 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:30:02.742213 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.742181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b6c0d-46cc-4fb8-97df-91323202eef9-maas-api-tls\") pod \"maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:02.744570 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:02.744550 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b6c0d-46cc-4fb8-97df-91323202eef9-maas-api-tls\") pod \"maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:03.009757 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:03.009724 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:03.148830 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:03.148751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq"] Jul 22 15:30:03.151317 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:30:03.151287 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533b6c0d_46cc_4fb8_97df_91323202eef9.slice/crio-c72314ba504b9a65a3dbfa2affa24c4a8f110a16f632c8729fd6227c0ea9a34b WatchSource:0}: Error finding container c72314ba504b9a65a3dbfa2affa24c4a8f110a16f632c8729fd6227c0ea9a34b: Status 404 returned error can't find the container with id c72314ba504b9a65a3dbfa2affa24c4a8f110a16f632c8729fd6227c0ea9a34b Jul 22 15:30:03.152832 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:03.152809 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 15:30:03.523159 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:03.523107 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" event={"ID":"533b6c0d-46cc-4fb8-97df-91323202eef9","Type":"ContainerStarted","Data":"c72314ba504b9a65a3dbfa2affa24c4a8f110a16f632c8729fd6227c0ea9a34b"} Jul 22 15:30:06.539053 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:06.539017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" event={"ID":"533b6c0d-46cc-4fb8-97df-91323202eef9","Type":"ContainerStarted","Data":"1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e"} Jul 22 15:30:06.539470 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:06.539151 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:06.558100 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:06.558056 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" podStartSLOduration=1.866434543 podStartE2EDuration="4.558043555s" totalImagesPullingTime="2.691609012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:30:06.557443462 +0000 UTC m=+1532.056343238" watchObservedRunningTime="2026-07-22 15:30:06.558043555 +0000 UTC m=+1532.056943309" Jul 22 15:30:09.471423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:09.471387 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:30:12.549952 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:12.549924 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:17.518087 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:17.517998 2580 patch_prober.go:28] interesting pod/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 22 15:30:17.518500 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:17.518083 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 22 15:30:17.589783 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:17.589750 2580 generic.go:356] "Generic (PLEG): container finished" podID="658ec699-801c-4f6d-b886-56984effd723" containerID="58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7" exitCode=0 Jul 22 15:30:17.589931 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:17.589796 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" event={"ID":"658ec699-801c-4f6d-b886-56984effd723","Type":"ContainerDied","Data":"58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7"} Jul 22 15:30:17.590232 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:17.590215 2580 scope.go:117] "RemoveContainer" containerID="58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7" Jul 22 15:30:17.591156 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:17.591099 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:30:17.591269 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:17.591189 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:30:17.591269 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:17.591220 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:30:18.595832 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:18.595801 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" event={"ID":"658ec699-801c-4f6d-b886-56984effd723","Type":"ContainerStarted","Data":"918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c"} Jul 22 15:30:19.005252 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:19.005149 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:30:19.005436 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:19.005418 2580 patch_prober.go:28] interesting pod/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 22 15:30:19.005484 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:19.005460 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 22 15:30:20.005007 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:20.004970 2580 patch_prober.go:28] interesting pod/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 22 15:30:20.005430 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:20.005025 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 22 15:30:21.005186 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:21.005149 2580 patch_prober.go:28] interesting pod/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 22 15:30:21.005562 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:21.005205 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 22 15:30:22.005169 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:22.005105 2580 patch_prober.go:28] interesting pod/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 22 15:30:22.005349 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:22.005189 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 22 15:30:23.016262 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:23.016214 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:30:23.016692 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:23.016517 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:30:23.017159 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:23.017140 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:30:38.200165 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.200111 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq"] Jul 22 15:30:38.200687 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.200333 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" podUID="533b6c0d-46cc-4fb8-97df-91323202eef9" containerName="maas-api" containerID="cri-o://1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e" gracePeriod=30 Jul 22 15:30:38.444425 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.444399 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:38.553563 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.553493 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm598\" (UniqueName: \"kubernetes.io/projected/533b6c0d-46cc-4fb8-97df-91323202eef9-kube-api-access-pm598\") pod \"533b6c0d-46cc-4fb8-97df-91323202eef9\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " Jul 22 15:30:38.553563 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.553551 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b6c0d-46cc-4fb8-97df-91323202eef9-maas-api-tls\") pod \"533b6c0d-46cc-4fb8-97df-91323202eef9\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " Jul 22 15:30:38.553741 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.553569 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b6c0d-46cc-4fb8-97df-91323202eef9-openshift-service-ca\") pod \"533b6c0d-46cc-4fb8-97df-91323202eef9\" (UID: \"533b6c0d-46cc-4fb8-97df-91323202eef9\") " Jul 22 15:30:38.553969 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.553935 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533b6c0d-46cc-4fb8-97df-91323202eef9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "533b6c0d-46cc-4fb8-97df-91323202eef9" (UID: "533b6c0d-46cc-4fb8-97df-91323202eef9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:30:38.555712 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.555681 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533b6c0d-46cc-4fb8-97df-91323202eef9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "533b6c0d-46cc-4fb8-97df-91323202eef9" (UID: "533b6c0d-46cc-4fb8-97df-91323202eef9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 15:30:38.555712 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.555700 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533b6c0d-46cc-4fb8-97df-91323202eef9-kube-api-access-pm598" (OuterVolumeSpecName: "kube-api-access-pm598") pod "533b6c0d-46cc-4fb8-97df-91323202eef9" (UID: "533b6c0d-46cc-4fb8-97df-91323202eef9"). InnerVolumeSpecName "kube-api-access-pm598". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:30:38.654361 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.654334 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm598\" (UniqueName: \"kubernetes.io/projected/533b6c0d-46cc-4fb8-97df-91323202eef9-kube-api-access-pm598\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:30:38.654361 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.654357 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b6c0d-46cc-4fb8-97df-91323202eef9-maas-api-tls\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:30:38.654503 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.654367 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b6c0d-46cc-4fb8-97df-91323202eef9-openshift-service-ca\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:30:38.683151 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.683101 2580 generic.go:356] "Generic (PLEG): container finished" podID="533b6c0d-46cc-4fb8-97df-91323202eef9" containerID="1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e" exitCode=0 Jul 22 15:30:38.683255 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.683168 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" event={"ID":"533b6c0d-46cc-4fb8-97df-91323202eef9","Type":"ContainerDied","Data":"1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e"} Jul 22 15:30:38.683255 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.683175 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" Jul 22 15:30:38.683255 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.683208 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq" event={"ID":"533b6c0d-46cc-4fb8-97df-91323202eef9","Type":"ContainerDied","Data":"c72314ba504b9a65a3dbfa2affa24c4a8f110a16f632c8729fd6227c0ea9a34b"} Jul 22 15:30:38.683255 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.683244 2580 scope.go:117] "RemoveContainer" containerID="1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e" Jul 22 15:30:38.692226 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.692208 2580 scope.go:117] "RemoveContainer" containerID="1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e" Jul 22 15:30:38.692506 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:30:38.692488 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e\": container with ID starting with 1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e not found: ID does not exist" containerID="1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e" Jul 22 15:30:38.692569 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.692518 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e"} err="failed to get container status \"1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e\": rpc error: code = NotFound desc = could not find container \"1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e\": container with ID starting with 1741b205f1f3853a12396eabf5b3fe24870cfb63aed1ef70f2a3a3b5c225cb9e not found: ID does not exist" Jul 22 15:30:38.705400 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.705378 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq"] Jul 22 15:30:38.709543 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:38.709523 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6b1b943-5f85bb5799-lqrfq"] Jul 22 15:30:39.053944 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:39.053904 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="533b6c0d-46cc-4fb8-97df-91323202eef9" path="/var/lib/kubelet/pods/533b6c0d-46cc-4fb8-97df-91323202eef9/volumes" Jul 22 15:30:40.873622 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:40.873587 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:30:42.793851 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:42.793820 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g"] Jul 22 15:30:42.794324 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:42.794046 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" containerID="cri-o://918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c" gracePeriod=30 Jul 22 15:30:43.880013 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:43.879982 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95"] Jul 22 15:30:43.880474 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:43.880460 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533b6c0d-46cc-4fb8-97df-91323202eef9" containerName="maas-api" Jul 22 15:30:43.880522 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:43.880477 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="533b6c0d-46cc-4fb8-97df-91323202eef9" containerName="maas-api" Jul 22 15:30:43.880582 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:43.880571 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="533b6c0d-46cc-4fb8-97df-91323202eef9" containerName="maas-api" Jul 22 15:30:43.885188 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:43.885165 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:43.889063 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:43.889039 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7148173f-gw-openshift-default-dockercfg-lht64\"" Jul 22 15:30:43.899447 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:43.899425 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95"] Jul 22 15:30:44.000526 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.000494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-token\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.000656 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.000529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-credential-socket\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.000656 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.000550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-data\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.000656 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.000581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7b5e61-c26b-4440-be56-d8e46de7de75-istiod-ca-cert\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.000775 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.000664 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-envoy\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.000775 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.000693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-workload-socket\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.000775 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.000713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-workload-certs\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.000775 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.000742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-podinfo\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.000911 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.000775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpjl6\" (UniqueName: \"kubernetes.io/projected/3e7b5e61-c26b-4440-be56-d8e46de7de75-kube-api-access-xpjl6\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.101460 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.101423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpjl6\" (UniqueName: \"kubernetes.io/projected/3e7b5e61-c26b-4440-be56-d8e46de7de75-kube-api-access-xpjl6\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.101640 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.101494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-token\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.101640 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.101520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-credential-socket\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.101640 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.101549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-data\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.101640 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.101585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7b5e61-c26b-4440-be56-d8e46de7de75-istiod-ca-cert\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.101640 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.101616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-envoy\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.101918 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.101645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-workload-socket\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.101918 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.101674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-workload-certs\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.101918 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.101708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-podinfo\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.102140 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.101948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-credential-socket\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.102140 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.102051 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-data\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.102140 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.102081 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-workload-socket\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.102275 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.102170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-workload-certs\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.102355 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.102335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7b5e61-c26b-4440-be56-d8e46de7de75-istiod-ca-cert\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.104027 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.103994 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-envoy\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.104284 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.104267 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-podinfo\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.109747 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.109720 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-token\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.109953 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.109934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpjl6\" (UniqueName: \"kubernetes.io/projected/3e7b5e61-c26b-4440-be56-d8e46de7de75-kube-api-access-xpjl6\") pod \"e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.195321 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.195259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:44.330181 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.330158 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95"] Jul 22 15:30:44.332220 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:30:44.332191 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7b5e61_c26b_4440_be56_d8e46de7de75.slice/crio-f498874694f38a9edbf1d169e191388d8ea7b2c88c1b018a4110e28990624567 WatchSource:0}: Error finding container f498874694f38a9edbf1d169e191388d8ea7b2c88c1b018a4110e28990624567: Status 404 returned error can't find the container with id f498874694f38a9edbf1d169e191388d8ea7b2c88c1b018a4110e28990624567 Jul 22 15:30:44.334299 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.334267 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:30:44.334363 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.334327 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:30:44.334363 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.334357 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:30:44.710940 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.710909 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" event={"ID":"3e7b5e61-c26b-4440-be56-d8e46de7de75","Type":"ContainerStarted","Data":"91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7"} Jul 22 15:30:44.711107 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.710958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" event={"ID":"3e7b5e61-c26b-4440-be56-d8e46de7de75","Type":"ContainerStarted","Data":"f498874694f38a9edbf1d169e191388d8ea7b2c88c1b018a4110e28990624567"} Jul 22 15:30:44.732991 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:44.732944 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" podStartSLOduration=1.732928391 podStartE2EDuration="1.732928391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:30:44.730680612 +0000 UTC m=+1570.229580410" watchObservedRunningTime="2026-07-22 15:30:44.732928391 +0000 UTC m=+1570.231828166" Jul 22 15:30:45.195578 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:45.195548 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:45.200151 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:45.200131 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:45.716645 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:45.716613 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:45.717965 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:45.717939 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:46.280051 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.280020 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w"] Jul 22 15:30:46.283798 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.283775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:46.286627 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.286605 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7148173f-serving-cert\"" Jul 22 15:30:46.286736 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.286623 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 22 15:30:46.286736 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.286675 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8q48q\"" Jul 22 15:30:46.286894 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.286882 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 15:30:46.293797 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.293770 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w"] Jul 22 15:30:46.320293 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.320268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-openshift-service-ca\") pod \"maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:46.320392 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.320308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-maas-api-tls\") pod \"maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:46.320433 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.320416 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8qr5\" (UniqueName: \"kubernetes.io/projected/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-kube-api-access-z8qr5\") pod \"maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:46.421025 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.420986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qr5\" (UniqueName: \"kubernetes.io/projected/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-kube-api-access-z8qr5\") pod \"maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:46.421218 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.421052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-openshift-service-ca\") pod \"maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:46.421218 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.421096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-maas-api-tls\") pod \"maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:46.421356 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:30:46.421249 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-serving-cert: secret "maas-api-e2e-migrate-7148173f-serving-cert" not found Jul 22 15:30:46.421356 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:30:46.421334 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-maas-api-tls podName:e0677efc-d9d7-476a-be19-5e5dd1a8dd6a nodeName:}" failed. No retries permitted until 2026-07-22 15:30:46.921313779 +0000 UTC m=+1572.420213536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-maas-api-tls") pod "maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" (UID: "e0677efc-d9d7-476a-be19-5e5dd1a8dd6a") : secret "maas-api-e2e-migrate-7148173f-serving-cert" not found Jul 22 15:30:46.421796 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.421774 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-openshift-service-ca\") pod \"maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:46.437686 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.437664 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8qr5\" (UniqueName: \"kubernetes.io/projected/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-kube-api-access-z8qr5\") pod \"maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:46.925605 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.925573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-maas-api-tls\") pod \"maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:46.927865 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:46.927846 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-maas-api-tls\") pod \"maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:47.197327 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:47.197244 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:47.323644 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:47.323615 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w"] Jul 22 15:30:47.325907 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:30:47.325881 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0677efc_d9d7_476a_be19_5e5dd1a8dd6a.slice/crio-d8ffb47c87f61380dd84c9705d84106790d6ae9cce069d8db3f8c449d1d4f413 WatchSource:0}: Error finding container d8ffb47c87f61380dd84c9705d84106790d6ae9cce069d8db3f8c449d1d4f413: Status 404 returned error can't find the container with id d8ffb47c87f61380dd84c9705d84106790d6ae9cce069d8db3f8c449d1d4f413 Jul 22 15:30:47.726371 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:47.726336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" event={"ID":"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a","Type":"ContainerStarted","Data":"d8ffb47c87f61380dd84c9705d84106790d6ae9cce069d8db3f8c449d1d4f413"} Jul 22 15:30:48.045504 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.045482 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:30:48.135743 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.135708 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-credential-socket\") pod \"658ec699-801c-4f6d-b886-56984effd723\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " Jul 22 15:30:48.135898 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.135755 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-istio-envoy\") pod \"658ec699-801c-4f6d-b886-56984effd723\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " Jul 22 15:30:48.135898 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.135785 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-workload-socket\") pod \"658ec699-801c-4f6d-b886-56984effd723\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " Jul 22 15:30:48.135898 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.135819 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-istio-data\") pod \"658ec699-801c-4f6d-b886-56984effd723\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " Jul 22 15:30:48.135898 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.135841 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658ec699-801c-4f6d-b886-56984effd723-istio-podinfo\") pod \"658ec699-801c-4f6d-b886-56984effd723\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " Jul 22 15:30:48.135898 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.135869 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv9fn\" (UniqueName: \"kubernetes.io/projected/658ec699-801c-4f6d-b886-56984effd723-kube-api-access-xv9fn\") pod \"658ec699-801c-4f6d-b886-56984effd723\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " Jul 22 15:30:48.136202 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.135900 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-workload-certs\") pod \"658ec699-801c-4f6d-b886-56984effd723\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " Jul 22 15:30:48.136202 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.135941 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658ec699-801c-4f6d-b886-56984effd723-istio-token\") pod \"658ec699-801c-4f6d-b886-56984effd723\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " Jul 22 15:30:48.136202 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.135976 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658ec699-801c-4f6d-b886-56984effd723-istiod-ca-cert\") pod \"658ec699-801c-4f6d-b886-56984effd723\" (UID: \"658ec699-801c-4f6d-b886-56984effd723\") " Jul 22 15:30:48.136202 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.135986 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "658ec699-801c-4f6d-b886-56984effd723" (UID: "658ec699-801c-4f6d-b886-56984effd723"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:30:48.136414 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.136208 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-credential-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:30:48.136414 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.136211 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "658ec699-801c-4f6d-b886-56984effd723" (UID: "658ec699-801c-4f6d-b886-56984effd723"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:30:48.136414 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.136381 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "658ec699-801c-4f6d-b886-56984effd723" (UID: "658ec699-801c-4f6d-b886-56984effd723"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:30:48.136628 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.136570 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658ec699-801c-4f6d-b886-56984effd723-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "658ec699-801c-4f6d-b886-56984effd723" (UID: "658ec699-801c-4f6d-b886-56984effd723"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:30:48.137258 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.137232 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-istio-data" (OuterVolumeSpecName: "istio-data") pod "658ec699-801c-4f6d-b886-56984effd723" (UID: "658ec699-801c-4f6d-b886-56984effd723"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:30:48.138310 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.138282 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658ec699-801c-4f6d-b886-56984effd723-kube-api-access-xv9fn" (OuterVolumeSpecName: "kube-api-access-xv9fn") pod "658ec699-801c-4f6d-b886-56984effd723" (UID: "658ec699-801c-4f6d-b886-56984effd723"). InnerVolumeSpecName "kube-api-access-xv9fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:30:48.138408 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.138322 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/658ec699-801c-4f6d-b886-56984effd723-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "658ec699-801c-4f6d-b886-56984effd723" (UID: "658ec699-801c-4f6d-b886-56984effd723"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 15:30:48.138408 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.138330 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "658ec699-801c-4f6d-b886-56984effd723" (UID: "658ec699-801c-4f6d-b886-56984effd723"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:30:48.138408 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.138385 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658ec699-801c-4f6d-b886-56984effd723-istio-token" (OuterVolumeSpecName: "istio-token") pod "658ec699-801c-4f6d-b886-56984effd723" (UID: "658ec699-801c-4f6d-b886-56984effd723"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:30:48.236844 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.236818 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658ec699-801c-4f6d-b886-56984effd723-istiod-ca-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:30:48.236844 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.236841 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-istio-envoy\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:30:48.236844 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.236850 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-workload-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:30:48.237027 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.236858 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-istio-data\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:30:48.237027 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.236868 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658ec699-801c-4f6d-b886-56984effd723-istio-podinfo\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:30:48.237027 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.236876 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv9fn\" (UniqueName: \"kubernetes.io/projected/658ec699-801c-4f6d-b886-56984effd723-kube-api-access-xv9fn\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:30:48.237027 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.236884 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658ec699-801c-4f6d-b886-56984effd723-workload-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:30:48.237027 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.236892 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658ec699-801c-4f6d-b886-56984effd723-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:30:48.733022 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.732987 2580 generic.go:356] "Generic (PLEG): container finished" podID="658ec699-801c-4f6d-b886-56984effd723" containerID="918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c" exitCode=0 Jul 22 15:30:48.733572 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.733031 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" Jul 22 15:30:48.733572 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.733081 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" event={"ID":"658ec699-801c-4f6d-b886-56984effd723","Type":"ContainerDied","Data":"918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c"} Jul 22 15:30:48.733572 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.733156 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g" event={"ID":"658ec699-801c-4f6d-b886-56984effd723","Type":"ContainerDied","Data":"00f42c600d98a2fc029d205a190aff71969400a21b172afffaef52e55e8ba294"} Jul 22 15:30:48.733572 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.733186 2580 scope.go:117] "RemoveContainer" containerID="918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c" Jul 22 15:30:48.734971 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.734910 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" event={"ID":"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a","Type":"ContainerStarted","Data":"c25d90029253ca76465fe2ebcccd668bf59500792acb71746e7afdfc59a2a620"} Jul 22 15:30:48.735107 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.735061 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:48.743486 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.743471 2580 scope.go:117] "RemoveContainer" containerID="58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7" Jul 22 15:30:48.751007 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.750988 2580 scope.go:117] "RemoveContainer" containerID="918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c" Jul 22 15:30:48.751279 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:30:48.751256 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c\": container with ID starting with 918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c not found: ID does not exist" containerID="918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c" Jul 22 15:30:48.751327 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.751290 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c"} err="failed to get container status \"918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c\": rpc error: code = NotFound desc = could not find container \"918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c\": container with ID starting with 918f1a332d151300e4c3cf18624075f85e1bd8d91f89bd3ff2d470cbfbc80a3c not found: ID does not exist" Jul 22 15:30:48.751327 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.751307 2580 scope.go:117] "RemoveContainer" containerID="58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7" Jul 22 15:30:48.751584 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:30:48.751561 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7\": container with ID starting with 58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7 not found: ID does not exist" containerID="58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7" Jul 22 15:30:48.751627 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.751588 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7"} err="failed to get container status \"58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7\": rpc error: code = NotFound desc = could not find container \"58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7\": container with ID starting with 58c49a924e23a5a9f620e807653236830b7434bf81e85277114f5087a88569d7 not found: ID does not exist" Jul 22 15:30:48.769002 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.768962 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" podStartSLOduration=2.236869217 podStartE2EDuration="2.768950449s" totalImagesPullingTime="532.081232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:30:48.755660571 +0000 UTC m=+1574.254560344" watchObservedRunningTime="2026-07-22 15:30:48.768950449 +0000 UTC m=+1574.267850254" Jul 22 15:30:48.769164 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.769143 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g"] Jul 22 15:30:48.773387 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:48.773367 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c6b1b943-openshift-default-598978b59-xtv9g"] Jul 22 15:30:49.053853 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:49.053782 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="658ec699-801c-4f6d-b886-56984effd723" path="/var/lib/kubelet/pods/658ec699-801c-4f6d-b886-56984effd723/volumes" Jul 22 15:30:49.837595 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:49.837561 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:30:54.749473 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:54.749445 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:30:57.781315 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:57.781288 2580 generic.go:356] "Generic (PLEG): container finished" podID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerID="91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7" exitCode=0 Jul 22 15:30:57.781717 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:57.781361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" event={"ID":"3e7b5e61-c26b-4440-be56-d8e46de7de75","Type":"ContainerDied","Data":"91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7"} Jul 22 15:30:57.781717 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:57.781705 2580 scope.go:117] "RemoveContainer" containerID="91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7" Jul 22 15:30:57.782629 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:57.782603 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:30:57.782721 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:57.782664 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:30:57.782721 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:57.782695 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:30:58.787985 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:58.787951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" event={"ID":"3e7b5e61-c26b-4440-be56-d8e46de7de75","Type":"ContainerStarted","Data":"8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf"} Jul 22 15:30:59.196084 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:59.196048 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:30:59.197268 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:59.197237 2580 patch_prober.go:28] interesting pod/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 22 15:30:59.197402 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:30:59.197297 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" podUID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 22 15:31:00.195888 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:00.195847 2580 patch_prober.go:28] interesting pod/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 22 15:31:00.196259 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:00.195922 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" podUID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 22 15:31:00.717322 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:00.717288 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:31:01.195883 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:01.195848 2580 patch_prober.go:28] interesting pod/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 22 15:31:01.196048 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:01.195910 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" podUID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 22 15:31:02.196152 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:02.196098 2580 patch_prober.go:28] interesting pod/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 22 15:31:02.196624 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:02.196177 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" podUID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 22 15:31:03.200100 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:03.200072 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:31:03.201098 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:03.201080 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:31:21.666581 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:21.666538 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w"] Jul 22 15:31:21.667139 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:21.666755 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" podUID="e0677efc-d9d7-476a-be19-5e5dd1a8dd6a" containerName="maas-api" containerID="cri-o://c25d90029253ca76465fe2ebcccd668bf59500792acb71746e7afdfc59a2a620" gracePeriod=30 Jul 22 15:31:21.887916 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:21.887884 2580 generic.go:356] "Generic (PLEG): container finished" podID="e0677efc-d9d7-476a-be19-5e5dd1a8dd6a" containerID="c25d90029253ca76465fe2ebcccd668bf59500792acb71746e7afdfc59a2a620" exitCode=0 Jul 22 15:31:21.888069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:21.887940 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" event={"ID":"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a","Type":"ContainerDied","Data":"c25d90029253ca76465fe2ebcccd668bf59500792acb71746e7afdfc59a2a620"} Jul 22 15:31:21.916617 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:21.916564 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:31:22.005713 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.005682 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-openshift-service-ca\") pod \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " Jul 22 15:31:22.005899 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.005794 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-maas-api-tls\") pod \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " Jul 22 15:31:22.005899 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.005824 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8qr5\" (UniqueName: \"kubernetes.io/projected/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-kube-api-access-z8qr5\") pod \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\" (UID: \"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a\") " Jul 22 15:31:22.006041 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.006017 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e0677efc-d9d7-476a-be19-5e5dd1a8dd6a" (UID: "e0677efc-d9d7-476a-be19-5e5dd1a8dd6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:31:22.007893 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.007869 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-kube-api-access-z8qr5" (OuterVolumeSpecName: "kube-api-access-z8qr5") pod "e0677efc-d9d7-476a-be19-5e5dd1a8dd6a" (UID: "e0677efc-d9d7-476a-be19-5e5dd1a8dd6a"). InnerVolumeSpecName "kube-api-access-z8qr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:31:22.007994 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.007926 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e0677efc-d9d7-476a-be19-5e5dd1a8dd6a" (UID: "e0677efc-d9d7-476a-be19-5e5dd1a8dd6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 15:31:22.106686 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.106657 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-maas-api-tls\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:22.106802 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.106688 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8qr5\" (UniqueName: \"kubernetes.io/projected/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-kube-api-access-z8qr5\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:22.106802 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.106703 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a-openshift-service-ca\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:22.893314 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.893277 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" event={"ID":"e0677efc-d9d7-476a-be19-5e5dd1a8dd6a","Type":"ContainerDied","Data":"d8ffb47c87f61380dd84c9705d84106790d6ae9cce069d8db3f8c449d1d4f413"} Jul 22 15:31:22.893314 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.893303 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w" Jul 22 15:31:22.893811 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.893347 2580 scope.go:117] "RemoveContainer" containerID="c25d90029253ca76465fe2ebcccd668bf59500792acb71746e7afdfc59a2a620" Jul 22 15:31:22.921072 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.921039 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w"] Jul 22 15:31:22.926174 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:22.926150 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7148173f-78cdc7ff4c-pvq7w"] Jul 22 15:31:23.054183 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:23.054146 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0677efc-d9d7-476a-be19-5e5dd1a8dd6a" path="/var/lib/kubelet/pods/e0677efc-d9d7-476a-be19-5e5dd1a8dd6a/volumes" Jul 22 15:31:24.472527 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:24.472493 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:31:26.475065 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:26.475031 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95"] Jul 22 15:31:26.475634 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:26.475239 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" podUID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerName="istio-proxy" containerID="cri-o://8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf" gracePeriod=30 Jul 22 15:31:27.309219 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.309184 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr"] Jul 22 15:31:27.309600 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.309582 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" Jul 22 15:31:27.309694 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.309602 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" Jul 22 15:31:27.309694 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.309624 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0677efc-d9d7-476a-be19-5e5dd1a8dd6a" containerName="maas-api" Jul 22 15:31:27.309694 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.309632 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0677efc-d9d7-476a-be19-5e5dd1a8dd6a" containerName="maas-api" Jul 22 15:31:27.309694 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.309646 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" Jul 22 15:31:27.309694 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.309655 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" Jul 22 15:31:27.309926 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.309769 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" Jul 22 15:31:27.309926 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.309786 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0677efc-d9d7-476a-be19-5e5dd1a8dd6a" containerName="maas-api" Jul 22 15:31:27.312810 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.312768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.315871 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.315824 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-17acd718-openshift-default-dockercfg-hstg5\"" Jul 22 15:31:27.326234 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.326208 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr"] Jul 22 15:31:27.453363 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.453326 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istiod-ca-cert\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.453557 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.453438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-data\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.453557 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.453507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-envoy\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.453557 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.453526 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-workload-certs\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.453557 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.453544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-workload-socket\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.453758 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.453560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-podinfo\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.453758 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.453606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-credential-socket\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.453758 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.453668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-token\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.453758 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.453693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxn7\" (UniqueName: \"kubernetes.io/projected/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-kube-api-access-gbxn7\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.554557 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.554523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-data\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.554988 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.554581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-envoy\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.554988 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.554695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-workload-certs\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.554988 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.554724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-workload-socket\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.554988 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.554748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-podinfo\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.554988 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.554782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-credential-socket\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.554988 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.554825 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-token\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.554988 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.554855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxn7\" (UniqueName: \"kubernetes.io/projected/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-kube-api-access-gbxn7\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.554988 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.554918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istiod-ca-cert\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.554988 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.554966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-data\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.555532 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.555109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-workload-socket\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.555532 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.555206 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-workload-certs\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.555532 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.555274 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-credential-socket\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.555532 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.555525 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istiod-ca-cert\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.557213 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.557186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-envoy\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.557359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.557326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-podinfo\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.563674 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.563595 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-token\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.563674 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.563640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxn7\" (UniqueName: \"kubernetes.io/projected/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-kube-api-access-gbxn7\") pod \"e2e-ait-17acd718-openshift-default-69597996b7-chplr\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.625841 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.625813 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:27.791797 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.791774 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr"] Jul 22 15:31:27.793472 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:31:27.793445 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee8e3cf_4dde_4f24_89e1_37e67d77d255.slice/crio-37cb1e575b9c056c87c1a287dbea66a9df602b905abde0674b0c71ac6b085010 WatchSource:0}: Error finding container 37cb1e575b9c056c87c1a287dbea66a9df602b905abde0674b0c71ac6b085010: Status 404 returned error can't find the container with id 37cb1e575b9c056c87c1a287dbea66a9df602b905abde0674b0c71ac6b085010 Jul 22 15:31:27.795562 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.795530 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:31:27.795675 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.795615 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:31:27.795675 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.795659 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:31:27.922033 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.921999 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" event={"ID":"7ee8e3cf-4dde-4f24-89e1-37e67d77d255","Type":"ContainerStarted","Data":"f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419"} Jul 22 15:31:27.922280 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.922071 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" event={"ID":"7ee8e3cf-4dde-4f24-89e1-37e67d77d255","Type":"ContainerStarted","Data":"37cb1e575b9c056c87c1a287dbea66a9df602b905abde0674b0c71ac6b085010"} Jul 22 15:31:27.949036 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:27.948979 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" podStartSLOduration=0.948958294 podStartE2EDuration="948.958294ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:31:27.946314929 +0000 UTC m=+1613.445214708" watchObservedRunningTime="2026-07-22 15:31:27.948958294 +0000 UTC m=+1613.447858070" Jul 22 15:31:28.626278 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:28.626246 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:28.631233 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:28.631210 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:28.926704 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:28.926631 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:28.927464 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:28.927447 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:31.724195 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.724169 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:31:31.892661 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.892629 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-credential-socket\") pod \"3e7b5e61-c26b-4440-be56-d8e46de7de75\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " Jul 22 15:31:31.892837 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.892690 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-workload-certs\") pod \"3e7b5e61-c26b-4440-be56-d8e46de7de75\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " Jul 22 15:31:31.892837 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.892747 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-envoy\") pod \"3e7b5e61-c26b-4440-be56-d8e46de7de75\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " Jul 22 15:31:31.892837 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.892775 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7b5e61-c26b-4440-be56-d8e46de7de75-istiod-ca-cert\") pod \"3e7b5e61-c26b-4440-be56-d8e46de7de75\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " Jul 22 15:31:31.892837 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.892803 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-workload-socket\") pod \"3e7b5e61-c26b-4440-be56-d8e46de7de75\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " Jul 22 15:31:31.893053 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.892840 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-token\") pod \"3e7b5e61-c26b-4440-be56-d8e46de7de75\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " Jul 22 15:31:31.893053 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.892864 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpjl6\" (UniqueName: \"kubernetes.io/projected/3e7b5e61-c26b-4440-be56-d8e46de7de75-kube-api-access-xpjl6\") pod \"3e7b5e61-c26b-4440-be56-d8e46de7de75\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " Jul 22 15:31:31.893053 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.892894 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-data\") pod \"3e7b5e61-c26b-4440-be56-d8e46de7de75\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " Jul 22 15:31:31.893053 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.892919 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-podinfo\") pod \"3e7b5e61-c26b-4440-be56-d8e46de7de75\" (UID: \"3e7b5e61-c26b-4440-be56-d8e46de7de75\") " Jul 22 15:31:31.893053 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.892938 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e7b5e61-c26b-4440-be56-d8e46de7de75" (UID: "3e7b5e61-c26b-4440-be56-d8e46de7de75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:31:31.893322 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.893061 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e7b5e61-c26b-4440-be56-d8e46de7de75" (UID: "3e7b5e61-c26b-4440-be56-d8e46de7de75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:31:31.893322 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.893177 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e7b5e61-c26b-4440-be56-d8e46de7de75" (UID: "3e7b5e61-c26b-4440-be56-d8e46de7de75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:31:31.893322 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.893180 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7b5e61-c26b-4440-be56-d8e46de7de75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e7b5e61-c26b-4440-be56-d8e46de7de75" (UID: "3e7b5e61-c26b-4440-be56-d8e46de7de75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:31:31.893322 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.893231 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-workload-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:31.893322 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.893254 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-credential-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:31.893756 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.893731 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e7b5e61-c26b-4440-be56-d8e46de7de75" (UID: "3e7b5e61-c26b-4440-be56-d8e46de7de75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:31:31.895032 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.895009 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e7b5e61-c26b-4440-be56-d8e46de7de75" (UID: "3e7b5e61-c26b-4440-be56-d8e46de7de75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:31:31.895172 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.895058 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e7b5e61-c26b-4440-be56-d8e46de7de75" (UID: "3e7b5e61-c26b-4440-be56-d8e46de7de75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 15:31:31.895453 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.895427 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e7b5e61-c26b-4440-be56-d8e46de7de75" (UID: "3e7b5e61-c26b-4440-be56-d8e46de7de75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:31:31.895522 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.895450 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7b5e61-c26b-4440-be56-d8e46de7de75-kube-api-access-xpjl6" (OuterVolumeSpecName: "kube-api-access-xpjl6") pod "3e7b5e61-c26b-4440-be56-d8e46de7de75" (UID: "3e7b5e61-c26b-4440-be56-d8e46de7de75"). InnerVolumeSpecName "kube-api-access-xpjl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:31:31.944515 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.944493 2580 generic.go:356] "Generic (PLEG): container finished" podID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerID="8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf" exitCode=0 Jul 22 15:31:31.944624 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.944548 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" Jul 22 15:31:31.944624 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.944558 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" event={"ID":"3e7b5e61-c26b-4440-be56-d8e46de7de75","Type":"ContainerDied","Data":"8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf"} Jul 22 15:31:31.944624 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.944600 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95" event={"ID":"3e7b5e61-c26b-4440-be56-d8e46de7de75","Type":"ContainerDied","Data":"f498874694f38a9edbf1d169e191388d8ea7b2c88c1b018a4110e28990624567"} Jul 22 15:31:31.944624 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.944623 2580 scope.go:117] "RemoveContainer" containerID="8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf" Jul 22 15:31:31.953926 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.953910 2580 scope.go:117] "RemoveContainer" containerID="91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7" Jul 22 15:31:31.961483 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.961465 2580 scope.go:117] "RemoveContainer" containerID="8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf" Jul 22 15:31:31.961726 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:31:31.961709 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf\": container with ID starting with 8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf not found: ID does not exist" containerID="8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf" Jul 22 15:31:31.961772 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.961734 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf"} err="failed to get container status \"8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf\": rpc error: code = NotFound desc = could not find container \"8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf\": container with ID starting with 8542f527e37278de022e9c01ad93c4746c335f2ac49ed81f6052562778887cbf not found: ID does not exist" Jul 22 15:31:31.961772 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.961750 2580 scope.go:117] "RemoveContainer" containerID="91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7" Jul 22 15:31:31.961989 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:31:31.961973 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7\": container with ID starting with 91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7 not found: ID does not exist" containerID="91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7" Jul 22 15:31:31.962029 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.961997 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7"} err="failed to get container status \"91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7\": rpc error: code = NotFound desc = could not find container \"91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7\": container with ID starting with 91b80c1ca1abdc148aa223c8eb9aea2083232f94083297e1800a1b14f6a83db7 not found: ID does not exist" Jul 22 15:31:31.967919 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.967899 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95"] Jul 22 15:31:31.971946 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.971924 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7148173f-gw-openshift-default-f765fcf75-jrk95"] Jul 22 15:31:31.994495 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.994474 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-envoy\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:31.994495 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.994493 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7b5e61-c26b-4440-be56-d8e46de7de75-istiod-ca-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:31.994597 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.994502 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-workload-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:31.994597 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.994510 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:31.994597 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.994519 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpjl6\" (UniqueName: \"kubernetes.io/projected/3e7b5e61-c26b-4440-be56-d8e46de7de75-kube-api-access-xpjl6\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:31.994597 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.994528 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-data\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:31.994597 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:31.994536 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7b5e61-c26b-4440-be56-d8e46de7de75-istio-podinfo\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:33.053778 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:33.053749 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e7b5e61-c26b-4440-be56-d8e46de7de75" path="/var/lib/kubelet/pods/3e7b5e61-c26b-4440-be56-d8e46de7de75/volumes" Jul 22 15:31:35.973910 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:35.972328 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:31:43.927921 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:43.927884 2580 patch_prober.go:28] interesting pod/e2e-ait-17acd718-openshift-default-69597996b7-chplr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 15:31:43.928333 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:43.927937 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 15:31:44.000544 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:44.000517 2580 generic.go:356] "Generic (PLEG): container finished" podID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerID="f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419" exitCode=0 Jul 22 15:31:44.000673 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:44.000592 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" event={"ID":"7ee8e3cf-4dde-4f24-89e1-37e67d77d255","Type":"ContainerDied","Data":"f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419"} Jul 22 15:31:44.000959 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:44.000944 2580 scope.go:117] "RemoveContainer" containerID="f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419" Jul 22 15:31:44.001948 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:44.001920 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:31:44.002019 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:44.002003 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:31:44.002068 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:44.002040 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:31:45.007390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:45.007354 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" event={"ID":"7ee8e3cf-4dde-4f24-89e1-37e67d77d255","Type":"ContainerStarted","Data":"9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa"} Jul 22 15:31:45.626426 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:45.626387 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:45.626623 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:45.626449 2580 patch_prober.go:28] interesting pod/e2e-ait-17acd718-openshift-default-69597996b7-chplr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 15:31:45.626623 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:45.626522 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 15:31:46.627079 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:46.626652 2580 patch_prober.go:28] interesting pod/e2e-ait-17acd718-openshift-default-69597996b7-chplr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 15:31:46.627079 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:46.626709 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 15:31:47.627227 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:47.627187 2580 patch_prober.go:28] interesting pod/e2e-ait-17acd718-openshift-default-69597996b7-chplr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 15:31:47.627698 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:47.627249 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 15:31:48.626532 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:48.626494 2580 patch_prober.go:28] interesting pod/e2e-ait-17acd718-openshift-default-69597996b7-chplr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 15:31:48.626814 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:48.626786 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 15:31:49.630444 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:49.630415 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:49.630833 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:49.630779 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:49.631545 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:49.631524 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:31:55.267804 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.267773 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c"] Jul 22 15:31:55.268209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.268076 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerName="istio-proxy" Jul 22 15:31:55.268209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.268086 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerName="istio-proxy" Jul 22 15:31:55.268209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.268162 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerName="istio-proxy" Jul 22 15:31:55.268209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.268171 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="658ec699-801c-4f6d-b886-56984effd723" containerName="istio-proxy" Jul 22 15:31:55.270742 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.270727 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c" Jul 22 15:31:55.273894 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.273874 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b79tk\"" Jul 22 15:31:55.273984 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.273912 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 15:31:55.278567 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.278548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c"] Jul 22 15:31:55.294313 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.294289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073ab5fd-7ae8-4317-91c5-bc752c7819c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-17acd718-njw9c\" (UID: \"073ab5fd-7ae8-4317-91c5-bc752c7819c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c" Jul 22 15:31:55.394984 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.394956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073ab5fd-7ae8-4317-91c5-bc752c7819c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-17acd718-njw9c\" (UID: \"073ab5fd-7ae8-4317-91c5-bc752c7819c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c" Jul 22 15:31:55.395533 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.395514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073ab5fd-7ae8-4317-91c5-bc752c7819c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-17acd718-njw9c\" (UID: \"073ab5fd-7ae8-4317-91c5-bc752c7819c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c" Jul 22 15:31:55.582261 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.582238 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c" Jul 22 15:31:55.711449 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:55.711426 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c"] Jul 22 15:31:55.713622 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:31:55.713597 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073ab5fd_7ae8_4317_91c5_bc752c7819c1.slice/crio-1f51bbf456341275ef5e43b100a6e31f8d5e04522a5281eb967e6e984e218b1c WatchSource:0}: Error finding container 1f51bbf456341275ef5e43b100a6e31f8d5e04522a5281eb967e6e984e218b1c: Status 404 returned error can't find the container with id 1f51bbf456341275ef5e43b100a6e31f8d5e04522a5281eb967e6e984e218b1c Jul 22 15:31:56.061865 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:56.061837 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c" event={"ID":"073ab5fd-7ae8-4317-91c5-bc752c7819c1","Type":"ContainerStarted","Data":"7d5a7ae721cdbdf17e2a6560e8bba7553fa981e1885d6109f8597433ec818c67"} Jul 22 15:31:56.061980 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:56.061887 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c" event={"ID":"073ab5fd-7ae8-4317-91c5-bc752c7819c1","Type":"ContainerStarted","Data":"1f51bbf456341275ef5e43b100a6e31f8d5e04522a5281eb967e6e984e218b1c"} Jul 22 15:31:56.079103 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:56.079052 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c" podStartSLOduration=1.079037917 podStartE2EDuration="1.079037917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:31:56.077103199 +0000 UTC m=+1641.576002978" watchObservedRunningTime="2026-07-22 15:31:56.079037917 +0000 UTC m=+1641.577937692" Jul 22 15:31:57.067603 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:57.067572 2580 generic.go:356] "Generic (PLEG): container finished" podID="073ab5fd-7ae8-4317-91c5-bc752c7819c1" containerID="7d5a7ae721cdbdf17e2a6560e8bba7553fa981e1885d6109f8597433ec818c67" exitCode=0 Jul 22 15:31:57.068059 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:57.067665 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c" event={"ID":"073ab5fd-7ae8-4317-91c5-bc752c7819c1","Type":"ContainerDied","Data":"7d5a7ae721cdbdf17e2a6560e8bba7553fa981e1885d6109f8597433ec818c67"} Jul 22 15:31:58.201201 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:58.201178 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c" Jul 22 15:31:58.321251 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:58.321215 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073ab5fd-7ae8-4317-91c5-bc752c7819c1-maas-api-service-ca\") pod \"073ab5fd-7ae8-4317-91c5-bc752c7819c1\" (UID: \"073ab5fd-7ae8-4317-91c5-bc752c7819c1\") " Jul 22 15:31:58.321568 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:58.321550 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073ab5fd-7ae8-4317-91c5-bc752c7819c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "073ab5fd-7ae8-4317-91c5-bc752c7819c1" (UID: "073ab5fd-7ae8-4317-91c5-bc752c7819c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:31:58.422740 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:58.422672 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073ab5fd-7ae8-4317-91c5-bc752c7819c1-maas-api-service-ca\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:31:59.077063 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:59.077034 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c" Jul 22 15:31:59.077208 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:59.077035 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c" event={"ID":"073ab5fd-7ae8-4317-91c5-bc752c7819c1","Type":"ContainerDied","Data":"1f51bbf456341275ef5e43b100a6e31f8d5e04522a5281eb967e6e984e218b1c"} Jul 22 15:31:59.077208 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:31:59.077173 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f51bbf456341275ef5e43b100a6e31f8d5e04522a5281eb967e6e984e218b1c" Jul 22 15:32:08.568544 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:08.568505 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:32:10.345165 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:10.345133 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c"] Jul 22 15:32:10.348560 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:10.348535 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17acd718-njw9c"] Jul 22 15:32:11.053961 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:11.053929 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="073ab5fd-7ae8-4317-91c5-bc752c7819c1" path="/var/lib/kubelet/pods/073ab5fd-7ae8-4317-91c5-bc752c7819c1/volumes" Jul 22 15:32:11.672779 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:11.672746 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr"] Jul 22 15:32:11.673173 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:11.673034 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerName="istio-proxy" containerID="cri-o://9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa" gracePeriod=30 Jul 22 15:32:12.568582 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.568553 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv"] Jul 22 15:32:12.568906 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.568894 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerName="istio-proxy" Jul 22 15:32:12.568949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.568920 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerName="istio-proxy" Jul 22 15:32:12.568949 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.568939 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="073ab5fd-7ae8-4317-91c5-bc752c7819c1" containerName="revoke-keys" Jul 22 15:32:12.569009 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.568948 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="073ab5fd-7ae8-4317-91c5-bc752c7819c1" containerName="revoke-keys" Jul 22 15:32:12.569041 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.569009 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7b5e61-c26b-4440-be56-d8e46de7de75" containerName="istio-proxy" Jul 22 15:32:12.569041 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.569025 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="073ab5fd-7ae8-4317-91c5-bc752c7819c1" containerName="revoke-keys" Jul 22 15:32:12.572070 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.572049 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.574820 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.574799 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5fe701b2-openshift-default-dockercfg-gs45q\"" Jul 22 15:32:12.584158 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.584134 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv"] Jul 22 15:32:12.644341 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.644308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-token\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.644496 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.644360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istiod-ca-cert\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.644496 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.644410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-envoy\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.644496 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.644427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-podinfo\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.644595 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.644498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp5qj\" (UniqueName: \"kubernetes.io/projected/3cc812c8-b15c-48b1-b214-52d9edcc1d14-kube-api-access-cp5qj\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.644595 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.644540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-workload-socket\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.644595 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.644559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-credential-socket\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.644595 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.644575 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-data\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.644819 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.644614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-workload-certs\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.745191 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp5qj\" (UniqueName: \"kubernetes.io/projected/3cc812c8-b15c-48b1-b214-52d9edcc1d14-kube-api-access-cp5qj\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.745191 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-workload-socket\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.745703 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-credential-socket\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.745703 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-data\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.745703 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-workload-certs\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.745703 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-token\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.745703 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istiod-ca-cert\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.745703 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-envoy\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.745703 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-credential-socket\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.745703 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-podinfo\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.746069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745754 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-data\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.746069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745903 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-workload-socket\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.746069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.745967 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-workload-certs\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.746367 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.746340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istiod-ca-cert\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.747902 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.747881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-envoy\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.747999 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.747967 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-podinfo\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.753778 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.753757 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-token\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.754017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.754000 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp5qj\" (UniqueName: \"kubernetes.io/projected/3cc812c8-b15c-48b1-b214-52d9edcc1d14-kube-api-access-cp5qj\") pod \"e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:12.882609 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:12.882529 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:13.019042 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:13.019018 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv"] Jul 22 15:32:13.021027 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:32:13.020995 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc812c8_b15c_48b1_b214_52d9edcc1d14.slice/crio-d3dce9c0231538a301edf944db04406b29aa9f30045f74e1bec55e41bb55ea16 WatchSource:0}: Error finding container d3dce9c0231538a301edf944db04406b29aa9f30045f74e1bec55e41bb55ea16: Status 404 returned error can't find the container with id d3dce9c0231538a301edf944db04406b29aa9f30045f74e1bec55e41bb55ea16 Jul 22 15:32:13.023920 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:13.023879 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:32:13.024032 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:13.023972 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:32:13.024032 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:13.024015 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:32:13.139525 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:13.139447 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" event={"ID":"3cc812c8-b15c-48b1-b214-52d9edcc1d14","Type":"ContainerStarted","Data":"c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a"} Jul 22 15:32:13.139525 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:13.139508 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" event={"ID":"3cc812c8-b15c-48b1-b214-52d9edcc1d14","Type":"ContainerStarted","Data":"d3dce9c0231538a301edf944db04406b29aa9f30045f74e1bec55e41bb55ea16"} Jul 22 15:32:13.159162 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:13.159091 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" podStartSLOduration=1.159071451 podStartE2EDuration="1.159071451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:32:13.15838434 +0000 UTC m=+1658.657284125" watchObservedRunningTime="2026-07-22 15:32:13.159071451 +0000 UTC m=+1658.657971228" Jul 22 15:32:13.882952 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:13.882918 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:14.888360 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:14.888328 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:15.149454 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:15.149375 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:15.150228 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:15.150206 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:16.931013 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:16.930990 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:32:17.084834 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.084792 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-credential-socket\") pod \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " Jul 22 15:32:17.084834 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.084838 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbxn7\" (UniqueName: \"kubernetes.io/projected/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-kube-api-access-gbxn7\") pod \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " Jul 22 15:32:17.085069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.084863 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-token\") pod \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " Jul 22 15:32:17.085069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.084926 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-workload-socket\") pod \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " Jul 22 15:32:17.085069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.084949 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istiod-ca-cert\") pod \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " Jul 22 15:32:17.085069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.084968 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-podinfo\") pod \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " Jul 22 15:32:17.085069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.085004 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-data\") pod \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " Jul 22 15:32:17.085069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.085030 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-envoy\") pod \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " Jul 22 15:32:17.085069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.085067 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-workload-certs\") pod \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\" (UID: \"7ee8e3cf-4dde-4f24-89e1-37e67d77d255\") " Jul 22 15:32:17.085453 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.085082 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ee8e3cf-4dde-4f24-89e1-37e67d77d255" (UID: "7ee8e3cf-4dde-4f24-89e1-37e67d77d255"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:32:17.085453 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.085207 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ee8e3cf-4dde-4f24-89e1-37e67d77d255" (UID: "7ee8e3cf-4dde-4f24-89e1-37e67d77d255"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:32:17.085453 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.085351 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-workload-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:32:17.085453 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.085365 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ee8e3cf-4dde-4f24-89e1-37e67d77d255" (UID: "7ee8e3cf-4dde-4f24-89e1-37e67d77d255"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:32:17.085453 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.085373 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-credential-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:32:17.085453 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.085429 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ee8e3cf-4dde-4f24-89e1-37e67d77d255" (UID: "7ee8e3cf-4dde-4f24-89e1-37e67d77d255"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:32:17.085813 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.085788 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ee8e3cf-4dde-4f24-89e1-37e67d77d255" (UID: "7ee8e3cf-4dde-4f24-89e1-37e67d77d255"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:32:17.087095 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.087075 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ee8e3cf-4dde-4f24-89e1-37e67d77d255" (UID: "7ee8e3cf-4dde-4f24-89e1-37e67d77d255"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:32:17.087210 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.087097 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ee8e3cf-4dde-4f24-89e1-37e67d77d255" (UID: "7ee8e3cf-4dde-4f24-89e1-37e67d77d255"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:32:17.087420 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.087404 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-kube-api-access-gbxn7" (OuterVolumeSpecName: "kube-api-access-gbxn7") pod "7ee8e3cf-4dde-4f24-89e1-37e67d77d255" (UID: "7ee8e3cf-4dde-4f24-89e1-37e67d77d255"). InnerVolumeSpecName "kube-api-access-gbxn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:32:17.087567 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.087547 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ee8e3cf-4dde-4f24-89e1-37e67d77d255" (UID: "7ee8e3cf-4dde-4f24-89e1-37e67d77d255"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 15:32:17.161253 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.161229 2580 generic.go:356] "Generic (PLEG): container finished" podID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerID="9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa" exitCode=0 Jul 22 15:32:17.161361 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.161282 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" Jul 22 15:32:17.161361 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.161310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" event={"ID":"7ee8e3cf-4dde-4f24-89e1-37e67d77d255","Type":"ContainerDied","Data":"9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa"} Jul 22 15:32:17.161465 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.161364 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr" event={"ID":"7ee8e3cf-4dde-4f24-89e1-37e67d77d255","Type":"ContainerDied","Data":"37cb1e575b9c056c87c1a287dbea66a9df602b905abde0674b0c71ac6b085010"} Jul 22 15:32:17.161465 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.161386 2580 scope.go:117] "RemoveContainer" containerID="9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa" Jul 22 15:32:17.170017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.170000 2580 scope.go:117] "RemoveContainer" containerID="f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419" Jul 22 15:32:17.179764 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.179744 2580 scope.go:117] "RemoveContainer" containerID="9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa" Jul 22 15:32:17.180041 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:32:17.180018 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa\": container with ID starting with 9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa not found: ID does not exist" containerID="9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa" Jul 22 15:32:17.180112 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.180040 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa"} err="failed to get container status \"9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa\": rpc error: code = NotFound desc = could not find container \"9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa\": container with ID starting with 9afb98aa9343aa0c4394b9746192a7d59dd4f87f9017382fffd06c3d66a1cfaa not found: ID does not exist" Jul 22 15:32:17.180112 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.180058 2580 scope.go:117] "RemoveContainer" containerID="f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419" Jul 22 15:32:17.180351 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:32:17.180327 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419\": container with ID starting with f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419 not found: ID does not exist" containerID="f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419" Jul 22 15:32:17.180426 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.180358 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419"} err="failed to get container status \"f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419\": rpc error: code = NotFound desc = could not find container \"f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419\": container with ID starting with f1906ad1bd5458efb02ac8e2fff6a239cf3b1ec7e1f8705250bb28eec8c22419 not found: ID does not exist" Jul 22 15:32:17.180709 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.180692 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:32:17.184912 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.184894 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr"] Jul 22 15:32:17.186475 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.186460 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-data\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:32:17.186538 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.186478 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-envoy\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:32:17.186538 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.186487 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-workload-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:32:17.186538 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.186496 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbxn7\" (UniqueName: \"kubernetes.io/projected/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-kube-api-access-gbxn7\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:32:17.186538 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.186505 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:32:17.186538 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.186515 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istiod-ca-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:32:17.186538 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.186526 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee8e3cf-4dde-4f24-89e1-37e67d77d255-istio-podinfo\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:32:17.193490 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:17.193466 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-17acd718-openshift-default-69597996b7-chplr"] Jul 22 15:32:19.053897 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:19.053855 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" path="/var/lib/kubelet/pods/7ee8e3cf-4dde-4f24-89e1-37e67d77d255/volumes" Jul 22 15:32:26.212855 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:26.212825 2580 generic.go:356] "Generic (PLEG): container finished" podID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerID="c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a" exitCode=0 Jul 22 15:32:26.213298 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:26.212897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" event={"ID":"3cc812c8-b15c-48b1-b214-52d9edcc1d14","Type":"ContainerDied","Data":"c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a"} Jul 22 15:32:26.213298 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:26.213278 2580 scope.go:117] "RemoveContainer" containerID="c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a" Jul 22 15:32:26.214359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:26.214327 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:32:26.214440 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:26.214413 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:32:26.214486 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:26.214460 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:32:27.221925 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:27.221883 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" event={"ID":"3cc812c8-b15c-48b1-b214-52d9edcc1d14","Type":"ContainerStarted","Data":"7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06"} Jul 22 15:32:27.883417 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:27.883382 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:27.884509 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:27.884487 2580 patch_prober.go:28] interesting pod/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 22 15:32:27.884563 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:27.884535 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" podUID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 22 15:32:28.888508 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:28.888469 2580 patch_prober.go:28] interesting pod/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 22 15:32:28.888969 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:28.888531 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" podUID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 22 15:32:29.883679 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:29.883645 2580 patch_prober.go:28] interesting pod/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 22 15:32:29.883854 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:29.883711 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" podUID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 22 15:32:30.149994 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:30.149909 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:30.883536 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:30.883497 2580 patch_prober.go:28] interesting pod/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 22 15:32:30.883746 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:30.883555 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" podUID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 22 15:32:31.887100 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:31.887067 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:31.888134 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:31.888100 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:32:53.369658 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:53.369627 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:32:54.683704 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:54.683676 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv"] Jul 22 15:32:54.684138 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:54.683874 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" podUID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerName="istio-proxy" containerID="cri-o://7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06" gracePeriod=30 Jul 22 15:32:59.946098 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:32:59.946037 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:33:00.023486 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.023454 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-podinfo\") pod \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " Jul 22 15:33:00.023660 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.023507 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-workload-certs\") pod \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " Jul 22 15:33:00.023660 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.023533 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-envoy\") pod \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " Jul 22 15:33:00.023660 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.023555 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istiod-ca-cert\") pod \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " Jul 22 15:33:00.023660 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.023580 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-credential-socket\") pod \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " Jul 22 15:33:00.023660 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.023615 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-data\") pod \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " Jul 22 15:33:00.023660 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.023631 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cp5qj\" (UniqueName: \"kubernetes.io/projected/3cc812c8-b15c-48b1-b214-52d9edcc1d14-kube-api-access-cp5qj\") pod \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " Jul 22 15:33:00.023956 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.023682 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-workload-socket\") pod \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " Jul 22 15:33:00.023956 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.023709 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-token\") pod \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\" (UID: \"3cc812c8-b15c-48b1-b214-52d9edcc1d14\") " Jul 22 15:33:00.023956 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.023796 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3cc812c8-b15c-48b1-b214-52d9edcc1d14" (UID: "3cc812c8-b15c-48b1-b214-52d9edcc1d14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:33:00.024110 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.023983 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3cc812c8-b15c-48b1-b214-52d9edcc1d14" (UID: "3cc812c8-b15c-48b1-b214-52d9edcc1d14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:33:00.024110 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.024003 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-workload-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:33:00.024110 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.024072 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3cc812c8-b15c-48b1-b214-52d9edcc1d14" (UID: "3cc812c8-b15c-48b1-b214-52d9edcc1d14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:33:00.024292 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.024230 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3cc812c8-b15c-48b1-b214-52d9edcc1d14" (UID: "3cc812c8-b15c-48b1-b214-52d9edcc1d14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:33:00.024656 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.024628 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-data" (OuterVolumeSpecName: "istio-data") pod "3cc812c8-b15c-48b1-b214-52d9edcc1d14" (UID: "3cc812c8-b15c-48b1-b214-52d9edcc1d14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:33:00.025731 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.025709 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3cc812c8-b15c-48b1-b214-52d9edcc1d14" (UID: "3cc812c8-b15c-48b1-b214-52d9edcc1d14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 15:33:00.025939 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.025920 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-token" (OuterVolumeSpecName: "istio-token") pod "3cc812c8-b15c-48b1-b214-52d9edcc1d14" (UID: "3cc812c8-b15c-48b1-b214-52d9edcc1d14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:33:00.026012 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.025990 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3cc812c8-b15c-48b1-b214-52d9edcc1d14" (UID: "3cc812c8-b15c-48b1-b214-52d9edcc1d14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:33:00.026077 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.026056 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc812c8-b15c-48b1-b214-52d9edcc1d14-kube-api-access-cp5qj" (OuterVolumeSpecName: "kube-api-access-cp5qj") pod "3cc812c8-b15c-48b1-b214-52d9edcc1d14" (UID: "3cc812c8-b15c-48b1-b214-52d9edcc1d14"). InnerVolumeSpecName "kube-api-access-cp5qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:33:00.124692 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.124661 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-workload-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:33:00.124692 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.124686 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:33:00.124923 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.124700 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-podinfo\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:33:00.124923 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.124723 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-envoy\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:33:00.124923 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.124734 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istiod-ca-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:33:00.124923 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.124742 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-credential-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:33:00.124923 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.124750 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc812c8-b15c-48b1-b214-52d9edcc1d14-istio-data\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:33:00.124923 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.124759 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp5qj\" (UniqueName: \"kubernetes.io/projected/3cc812c8-b15c-48b1-b214-52d9edcc1d14-kube-api-access-cp5qj\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:33:00.374481 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.374446 2580 generic.go:356] "Generic (PLEG): container finished" podID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerID="7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06" exitCode=0 Jul 22 15:33:00.374631 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.374487 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" event={"ID":"3cc812c8-b15c-48b1-b214-52d9edcc1d14","Type":"ContainerDied","Data":"7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06"} Jul 22 15:33:00.374631 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.374499 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" Jul 22 15:33:00.374631 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.374538 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv" event={"ID":"3cc812c8-b15c-48b1-b214-52d9edcc1d14","Type":"ContainerDied","Data":"d3dce9c0231538a301edf944db04406b29aa9f30045f74e1bec55e41bb55ea16"} Jul 22 15:33:00.374631 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.374569 2580 scope.go:117] "RemoveContainer" containerID="7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06" Jul 22 15:33:00.383226 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.383201 2580 scope.go:117] "RemoveContainer" containerID="c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a" Jul 22 15:33:00.390557 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.390535 2580 scope.go:117] "RemoveContainer" containerID="7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06" Jul 22 15:33:00.390817 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:33:00.390798 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06\": container with ID starting with 7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06 not found: ID does not exist" containerID="7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06" Jul 22 15:33:00.390873 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.390826 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06"} err="failed to get container status \"7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06\": rpc error: code = NotFound desc = could not find container \"7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06\": container with ID starting with 7458f8c324a1c14d4c423808f3ca3c3feb98ae4dc0595b06b49f6581d62d0e06 not found: ID does not exist" Jul 22 15:33:00.390873 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.390842 2580 scope.go:117] "RemoveContainer" containerID="c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a" Jul 22 15:33:00.391078 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:33:00.391058 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a\": container with ID starting with c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a not found: ID does not exist" containerID="c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a" Jul 22 15:33:00.391153 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.391088 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a"} err="failed to get container status \"c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a\": rpc error: code = NotFound desc = could not find container \"c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a\": container with ID starting with c61ab7016a9fb3a96b1c361d817f19c74c86410f8b9bb41b470d7ad6f53abc3a not found: ID does not exist" Jul 22 15:33:00.396204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.396185 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv"] Jul 22 15:33:00.403331 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:00.403309 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5fe701b2-openshift-default-59fdc4bb99-9qnqv"] Jul 22 15:33:01.053403 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:33:01.053371 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" path="/var/lib/kubelet/pods/3cc812c8-b15c-48b1-b214-52d9edcc1d14/volumes" Jul 22 15:34:19.682101 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:19.682062 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:34:31.882791 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:31.882714 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:34:32.220299 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.220224 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw"] Jul 22 15:34:32.220611 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.220594 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerName="istio-proxy" Jul 22 15:34:32.220693 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.220614 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerName="istio-proxy" Jul 22 15:34:32.220693 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.220634 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerName="istio-proxy" Jul 22 15:34:32.220693 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.220646 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerName="istio-proxy" Jul 22 15:34:32.220693 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.220675 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerName="istio-proxy" Jul 22 15:34:32.220693 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.220683 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerName="istio-proxy" Jul 22 15:34:32.220946 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.220694 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerName="istio-proxy" Jul 22 15:34:32.220946 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.220703 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerName="istio-proxy" Jul 22 15:34:32.220946 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.220784 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerName="istio-proxy" Jul 22 15:34:32.220946 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.220796 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerName="istio-proxy" Jul 22 15:34:32.220946 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.220805 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cc812c8-b15c-48b1-b214-52d9edcc1d14" containerName="istio-proxy" Jul 22 15:34:32.223754 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.223733 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.226657 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.226640 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-41296eff-openshift-default-dockercfg-bshxw\"" Jul 22 15:34:32.250467 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.250447 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw"] Jul 22 15:34:32.349835 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.349805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmj94\" (UniqueName: \"kubernetes.io/projected/16975061-6a7f-4e77-9cbc-cd17bc7162c2-kube-api-access-xmj94\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.349835 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.349838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-data\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.350015 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.349861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istiod-ca-cert\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.350015 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.349888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-workload-socket\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.350015 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.349969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-workload-certs\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.350015 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.350001 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-token\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.350168 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.350033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-envoy\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.350168 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.350061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-credential-socket\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.350168 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.350079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-podinfo\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.450970 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.450933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-envoy\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451181 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.450976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-credential-socket\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451181 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.451002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-podinfo\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451181 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.451045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmj94\" (UniqueName: \"kubernetes.io/projected/16975061-6a7f-4e77-9cbc-cd17bc7162c2-kube-api-access-xmj94\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451181 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.451072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-data\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451181 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.451105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istiod-ca-cert\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451466 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.451418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-workload-socket\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451522 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.451487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-credential-socket\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451575 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.451516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-workload-certs\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451575 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.451562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-token\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451872 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.451849 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istiod-ca-cert\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451965 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.451880 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-workload-socket\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451965 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.451852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-data\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.451965 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.451915 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-workload-certs\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.453796 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.453774 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-envoy\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.454019 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.453997 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-podinfo\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.460759 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.460737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-token\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.461161 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.461140 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmj94\" (UniqueName: \"kubernetes.io/projected/16975061-6a7f-4e77-9cbc-cd17bc7162c2-kube-api-access-xmj94\") pod \"e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.533969 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.533912 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:32.691726 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.691561 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw"] Jul 22 15:34:32.694894 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:34:32.694857 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16975061_6a7f_4e77_9cbc_cd17bc7162c2.slice/crio-cfb60227ace2658535b722541db0a6c75db2d46f0be217c6999423d10de96ad6 WatchSource:0}: Error finding container cfb60227ace2658535b722541db0a6c75db2d46f0be217c6999423d10de96ad6: Status 404 returned error can't find the container with id cfb60227ace2658535b722541db0a6c75db2d46f0be217c6999423d10de96ad6 Jul 22 15:34:32.697256 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.697221 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:34:32.697342 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.697291 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:34:32.697342 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.697320 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:34:32.783256 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.783226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" event={"ID":"16975061-6a7f-4e77-9cbc-cd17bc7162c2","Type":"ContainerStarted","Data":"d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb"} Jul 22 15:34:32.783395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.783281 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" event={"ID":"16975061-6a7f-4e77-9cbc-cd17bc7162c2","Type":"ContainerStarted","Data":"cfb60227ace2658535b722541db0a6c75db2d46f0be217c6999423d10de96ad6"} Jul 22 15:34:32.811883 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:32.811771 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" podStartSLOduration=0.811751667 podStartE2EDuration="811.751667ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:34:32.811471523 +0000 UTC m=+1798.310371300" watchObservedRunningTime="2026-07-22 15:34:32.811751667 +0000 UTC m=+1798.310651445" Jul 22 15:34:33.534418 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:33.534387 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:33.539262 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:33.539239 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:33.788876 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:33.788803 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:33.789663 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:33.789646 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:43.469560 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:43.469522 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:34:47.508859 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:47.508824 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t"] Jul 22 15:34:47.509254 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:47.509201 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee8e3cf-4dde-4f24-89e1-37e67d77d255" containerName="istio-proxy" Jul 22 15:34:47.512521 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:47.512497 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t" Jul 22 15:34:47.516028 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:47.516007 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 15:34:47.516375 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:47.516065 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b79tk\"" Jul 22 15:34:47.520073 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:47.520042 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t"] Jul 22 15:34:47.673795 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:47.673739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a71533-20fc-467f-8990-dcf7a48bfb3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-41296eff-m6c5t\" (UID: \"c5a71533-20fc-467f-8990-dcf7a48bfb3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t" Jul 22 15:34:47.775201 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:47.775081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a71533-20fc-467f-8990-dcf7a48bfb3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-41296eff-m6c5t\" (UID: \"c5a71533-20fc-467f-8990-dcf7a48bfb3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t" Jul 22 15:34:47.775778 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:47.775753 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a71533-20fc-467f-8990-dcf7a48bfb3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-41296eff-m6c5t\" (UID: \"c5a71533-20fc-467f-8990-dcf7a48bfb3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t" Jul 22 15:34:47.829812 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:47.829778 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t" Jul 22 15:34:47.990948 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:47.990922 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t"] Jul 22 15:34:47.993264 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:34:47.993237 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a71533_20fc_467f_8990_dcf7a48bfb3a.slice/crio-68a52eceb13c8455b0ae6545c5ff145965f59a9091743cf9e253c55a6a0d1f52 WatchSource:0}: Error finding container 68a52eceb13c8455b0ae6545c5ff145965f59a9091743cf9e253c55a6a0d1f52: Status 404 returned error can't find the container with id 68a52eceb13c8455b0ae6545c5ff145965f59a9091743cf9e253c55a6a0d1f52 Jul 22 15:34:48.863940 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:48.863904 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t" event={"ID":"c5a71533-20fc-467f-8990-dcf7a48bfb3a","Type":"ContainerStarted","Data":"10b996cc49d412c696ca81fcb86316b84c2c62590d0d6ee38cf256b328564f7b"} Jul 22 15:34:48.864378 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:48.863956 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t" event={"ID":"c5a71533-20fc-467f-8990-dcf7a48bfb3a","Type":"ContainerStarted","Data":"68a52eceb13c8455b0ae6545c5ff145965f59a9091743cf9e253c55a6a0d1f52"} Jul 22 15:34:48.889303 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:48.889258 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t" podStartSLOduration=1.889243827 podStartE2EDuration="1.889243827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:34:48.887466195 +0000 UTC m=+1814.386365982" watchObservedRunningTime="2026-07-22 15:34:48.889243827 +0000 UTC m=+1814.388143601" Jul 22 15:34:49.831294 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:49.831251 2580 patch_prober.go:28] interesting pod/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 15:34:49.831530 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:49.831323 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 15:34:49.870518 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:49.870490 2580 generic.go:356] "Generic (PLEG): container finished" podID="c5a71533-20fc-467f-8990-dcf7a48bfb3a" containerID="10b996cc49d412c696ca81fcb86316b84c2c62590d0d6ee38cf256b328564f7b" exitCode=0 Jul 22 15:34:49.870920 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:49.870580 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t" event={"ID":"c5a71533-20fc-467f-8990-dcf7a48bfb3a","Type":"ContainerDied","Data":"10b996cc49d412c696ca81fcb86316b84c2c62590d0d6ee38cf256b328564f7b"} Jul 22 15:34:51.005850 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.005827 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t" Jul 22 15:34:51.103351 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.103321 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a71533-20fc-467f-8990-dcf7a48bfb3a-maas-api-service-ca\") pod \"c5a71533-20fc-467f-8990-dcf7a48bfb3a\" (UID: \"c5a71533-20fc-467f-8990-dcf7a48bfb3a\") " Jul 22 15:34:51.103718 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.103687 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a71533-20fc-467f-8990-dcf7a48bfb3a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5a71533-20fc-467f-8990-dcf7a48bfb3a" (UID: "c5a71533-20fc-467f-8990-dcf7a48bfb3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:34:51.204424 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.204339 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a71533-20fc-467f-8990-dcf7a48bfb3a-maas-api-service-ca\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:34:51.881084 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.881061 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t" Jul 22 15:34:51.881299 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.881099 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t" event={"ID":"c5a71533-20fc-467f-8990-dcf7a48bfb3a","Type":"ContainerDied","Data":"68a52eceb13c8455b0ae6545c5ff145965f59a9091743cf9e253c55a6a0d1f52"} Jul 22 15:34:51.881299 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.881163 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68a52eceb13c8455b0ae6545c5ff145965f59a9091743cf9e253c55a6a0d1f52" Jul 22 15:34:51.882601 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.882581 2580 generic.go:356] "Generic (PLEG): container finished" podID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerID="d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb" exitCode=0 Jul 22 15:34:51.882715 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.882614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" event={"ID":"16975061-6a7f-4e77-9cbc-cd17bc7162c2","Type":"ContainerDied","Data":"d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb"} Jul 22 15:34:51.882938 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.882923 2580 scope.go:117] "RemoveContainer" containerID="d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb" Jul 22 15:34:51.883883 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.883857 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:34:51.883953 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.883939 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:34:51.883990 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:51.883969 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:34:52.889705 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:52.889672 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" event={"ID":"16975061-6a7f-4e77-9cbc-cd17bc7162c2","Type":"ContainerStarted","Data":"062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b"} Jul 22 15:34:53.534848 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:53.534805 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:53.535064 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:53.534929 2580 patch_prober.go:28] interesting pod/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 22 15:34:53.535064 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:53.534973 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 22 15:34:54.535339 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:54.534863 2580 patch_prober.go:28] interesting pod/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 22 15:34:54.535339 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:54.534928 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 22 15:34:55.535259 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:55.535214 2580 patch_prober.go:28] interesting pod/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 22 15:34:55.535479 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:55.535282 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 22 15:34:56.534635 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:56.534589 2580 patch_prober.go:28] interesting pod/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 22 15:34:56.534833 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:56.534655 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 22 15:34:57.538634 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:57.538606 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:57.538999 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:57.538869 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:34:57.539608 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:34:57.539589 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:35:00.974477 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:00.974445 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:35:02.584668 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:02.584634 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t"] Jul 22 15:35:02.587749 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:02.587721 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-41296eff-m6c5t"] Jul 22 15:35:03.053812 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:03.053739 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a71533-20fc-467f-8990-dcf7a48bfb3a" path="/var/lib/kubelet/pods/c5a71533-20fc-467f-8990-dcf7a48bfb3a/volumes" Jul 22 15:35:09.159188 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:09.159151 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw"] Jul 22 15:35:09.159630 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:09.159385 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" containerID="cri-o://062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b" gracePeriod=30 Jul 22 15:35:12.541021 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:12.540979 2580 patch_prober.go:28] interesting pod/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 15:35:12.541401 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:12.541048 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 15:35:14.405922 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.405898 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:35:14.495111 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495044 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-token\") pod \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " Jul 22 15:35:14.495111 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495076 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istiod-ca-cert\") pod \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " Jul 22 15:35:14.495111 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495109 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-envoy\") pod \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " Jul 22 15:35:14.495395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495143 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-data\") pod \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " Jul 22 15:35:14.495395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495168 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-workload-socket\") pod \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " Jul 22 15:35:14.495395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495187 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-podinfo\") pod \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " Jul 22 15:35:14.495395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495209 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmj94\" (UniqueName: \"kubernetes.io/projected/16975061-6a7f-4e77-9cbc-cd17bc7162c2-kube-api-access-xmj94\") pod \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " Jul 22 15:35:14.495395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495237 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-workload-certs\") pod \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " Jul 22 15:35:14.495395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495271 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-credential-socket\") pod \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\" (UID: \"16975061-6a7f-4e77-9cbc-cd17bc7162c2\") " Jul 22 15:35:14.495685 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495522 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "16975061-6a7f-4e77-9cbc-cd17bc7162c2" (UID: "16975061-6a7f-4e77-9cbc-cd17bc7162c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:35:14.495727 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495678 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "16975061-6a7f-4e77-9cbc-cd17bc7162c2" (UID: "16975061-6a7f-4e77-9cbc-cd17bc7162c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:35:14.495774 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495731 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "16975061-6a7f-4e77-9cbc-cd17bc7162c2" (UID: "16975061-6a7f-4e77-9cbc-cd17bc7162c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:35:14.495816 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.495798 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "16975061-6a7f-4e77-9cbc-cd17bc7162c2" (UID: "16975061-6a7f-4e77-9cbc-cd17bc7162c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:35:14.496088 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.496042 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-data" (OuterVolumeSpecName: "istio-data") pod "16975061-6a7f-4e77-9cbc-cd17bc7162c2" (UID: "16975061-6a7f-4e77-9cbc-cd17bc7162c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:35:14.497412 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.497390 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "16975061-6a7f-4e77-9cbc-cd17bc7162c2" (UID: "16975061-6a7f-4e77-9cbc-cd17bc7162c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:35:14.497506 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.497393 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "16975061-6a7f-4e77-9cbc-cd17bc7162c2" (UID: "16975061-6a7f-4e77-9cbc-cd17bc7162c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:35:14.497506 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.497486 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16975061-6a7f-4e77-9cbc-cd17bc7162c2-kube-api-access-xmj94" (OuterVolumeSpecName: "kube-api-access-xmj94") pod "16975061-6a7f-4e77-9cbc-cd17bc7162c2" (UID: "16975061-6a7f-4e77-9cbc-cd17bc7162c2"). InnerVolumeSpecName "kube-api-access-xmj94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:35:14.497610 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.497578 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "16975061-6a7f-4e77-9cbc-cd17bc7162c2" (UID: "16975061-6a7f-4e77-9cbc-cd17bc7162c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 15:35:14.596125 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.596100 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-credential-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:35:14.596242 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.596145 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:35:14.596242 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.596157 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istiod-ca-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:35:14.596242 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.596165 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-envoy\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:35:14.596242 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.596172 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-data\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:35:14.596242 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.596180 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-workload-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:35:14.596242 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.596187 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16975061-6a7f-4e77-9cbc-cd17bc7162c2-istio-podinfo\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:35:14.596242 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.596195 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmj94\" (UniqueName: \"kubernetes.io/projected/16975061-6a7f-4e77-9cbc-cd17bc7162c2-kube-api-access-xmj94\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:35:14.596242 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.596203 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16975061-6a7f-4e77-9cbc-cd17bc7162c2-workload-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:35:14.990926 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.990884 2580 generic.go:356] "Generic (PLEG): container finished" podID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerID="062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b" exitCode=0 Jul 22 15:35:14.991146 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.990933 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" Jul 22 15:35:14.991146 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.990971 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" event={"ID":"16975061-6a7f-4e77-9cbc-cd17bc7162c2","Type":"ContainerDied","Data":"062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b"} Jul 22 15:35:14.991146 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.991036 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw" event={"ID":"16975061-6a7f-4e77-9cbc-cd17bc7162c2","Type":"ContainerDied","Data":"cfb60227ace2658535b722541db0a6c75db2d46f0be217c6999423d10de96ad6"} Jul 22 15:35:14.991146 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:14.991068 2580 scope.go:117] "RemoveContainer" containerID="062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b" Jul 22 15:35:15.000518 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:15.000497 2580 scope.go:117] "RemoveContainer" containerID="d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb" Jul 22 15:35:15.008453 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:15.008432 2580 scope.go:117] "RemoveContainer" containerID="062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b" Jul 22 15:35:15.008688 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:35:15.008670 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b\": container with ID starting with 062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b not found: ID does not exist" containerID="062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b" Jul 22 15:35:15.008749 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:15.008692 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b"} err="failed to get container status \"062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b\": rpc error: code = NotFound desc = could not find container \"062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b\": container with ID starting with 062c271401bfc95f1c60c43e66e4f3c6a179c2c9e22f55f7658dcbae5c46c75b not found: ID does not exist" Jul 22 15:35:15.008749 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:15.008707 2580 scope.go:117] "RemoveContainer" containerID="d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb" Jul 22 15:35:15.008933 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:35:15.008916 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb\": container with ID starting with d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb not found: ID does not exist" containerID="d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb" Jul 22 15:35:15.008975 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:15.008939 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb"} err="failed to get container status \"d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb\": rpc error: code = NotFound desc = could not find container \"d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb\": container with ID starting with d47325205e3c7d877a7ef674c747f49b5f05762bf276921fcec50c64a3847deb not found: ID does not exist" Jul 22 15:35:15.013581 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:15.013560 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw"] Jul 22 15:35:15.020270 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:15.020250 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-41296eff-openshift-default-7966bd5cdc-2rskw"] Jul 22 15:35:15.053799 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:15.053776 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" path="/var/lib/kubelet/pods/16975061-6a7f-4e77-9cbc-cd17bc7162c2/volumes" Jul 22 15:35:29.118839 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.118807 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7"] Jul 22 15:35:29.119314 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.119147 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a71533-20fc-467f-8990-dcf7a48bfb3a" containerName="revoke-keys" Jul 22 15:35:29.119314 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.119159 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a71533-20fc-467f-8990-dcf7a48bfb3a" containerName="revoke-keys" Jul 22 15:35:29.119314 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.119167 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" Jul 22 15:35:29.119314 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.119172 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" Jul 22 15:35:29.119314 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.119181 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" Jul 22 15:35:29.119314 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.119186 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" Jul 22 15:35:29.119314 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.119242 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" Jul 22 15:35:29.119314 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.119249 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a71533-20fc-467f-8990-dcf7a48bfb3a" containerName="revoke-keys" Jul 22 15:35:29.119314 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.119256 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="16975061-6a7f-4e77-9cbc-cd17bc7162c2" containerName="istio-proxy" Jul 22 15:35:29.121994 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.121979 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.124979 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.124958 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ca366ee0-openshift-default-dockercfg-zxxbm\"" Jul 22 15:35:29.133037 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.133013 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7"] Jul 22 15:35:29.218762 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.218725 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-698pp\" (UniqueName: \"kubernetes.io/projected/89f4bccb-46f7-4945-890c-a64eb25a073f-kube-api-access-698pp\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.219017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.218995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-podinfo\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.219166 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.219148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-token\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.219329 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.219313 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-workload-socket\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.219443 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.219429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-envoy\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.219568 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.219545 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-data\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.219698 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.219607 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-credential-socket\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.219698 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.219637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-workload-certs\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.219698 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.219691 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f4bccb-46f7-4945-890c-a64eb25a073f-istiod-ca-cert\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.320422 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.320380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-698pp\" (UniqueName: \"kubernetes.io/projected/89f4bccb-46f7-4945-890c-a64eb25a073f-kube-api-access-698pp\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.320615 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.320434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-podinfo\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.320615 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.320565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-token\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.320727 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.320704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-workload-socket\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.320785 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.320733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-envoy\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.320785 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.320770 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-data\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.320885 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.320816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-credential-socket\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.320885 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.320843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-workload-certs\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.320994 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.320903 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f4bccb-46f7-4945-890c-a64eb25a073f-istiod-ca-cert\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.321176 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.321149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-workload-socket\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.321513 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.321487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-data\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.321658 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.321517 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-credential-socket\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.321708 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.321654 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-workload-certs\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.321708 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.321658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f4bccb-46f7-4945-890c-a64eb25a073f-istiod-ca-cert\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.323427 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.323399 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-envoy\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.323516 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.323472 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-podinfo\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.329755 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.329730 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-token\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.329902 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.329884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-698pp\" (UniqueName: \"kubernetes.io/projected/89f4bccb-46f7-4945-890c-a64eb25a073f-kube-api-access-698pp\") pod \"e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.435198 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.435103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:29.568395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.568367 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7"] Jul 22 15:35:29.570812 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:35:29.570782 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f4bccb_46f7_4945_890c_a64eb25a073f.slice/crio-fcdc8b68bcd4982e4e74fcf500fb1cfe6da018c7f63ce73f794136a36034e9e6 WatchSource:0}: Error finding container fcdc8b68bcd4982e4e74fcf500fb1cfe6da018c7f63ce73f794136a36034e9e6: Status 404 returned error can't find the container with id fcdc8b68bcd4982e4e74fcf500fb1cfe6da018c7f63ce73f794136a36034e9e6 Jul 22 15:35:29.572869 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.572837 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:35:29.572970 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.572912 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:35:29.572970 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:29.572953 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:35:30.068086 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:30.068048 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" event={"ID":"89f4bccb-46f7-4945-890c-a64eb25a073f","Type":"ContainerStarted","Data":"849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7"} Jul 22 15:35:30.068270 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:30.068113 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" event={"ID":"89f4bccb-46f7-4945-890c-a64eb25a073f","Type":"ContainerStarted","Data":"fcdc8b68bcd4982e4e74fcf500fb1cfe6da018c7f63ce73f794136a36034e9e6"} Jul 22 15:35:30.093383 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:30.093328 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" podStartSLOduration=1.093309202 podStartE2EDuration="1.093309202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:35:30.090495726 +0000 UTC m=+1855.589395503" watchObservedRunningTime="2026-07-22 15:35:30.093309202 +0000 UTC m=+1855.592208982" Jul 22 15:35:30.435890 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:30.435811 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:30.437217 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:30.437183 2580 patch_prober.go:28] interesting pod/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 22 15:35:30.437345 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:30.437237 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 22 15:35:31.439843 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:31.439807 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:32.076167 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:32.076137 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:32.077059 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:32.077043 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:36.769520 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:36.769484 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:35:45.139794 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:45.139762 2580 generic.go:356] "Generic (PLEG): container finished" podID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerID="849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7" exitCode=0 Jul 22 15:35:45.140342 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:45.139841 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" event={"ID":"89f4bccb-46f7-4945-890c-a64eb25a073f","Type":"ContainerDied","Data":"849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7"} Jul 22 15:35:45.140418 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:45.140340 2580 scope.go:117] "RemoveContainer" containerID="849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7" Jul 22 15:35:45.141341 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:45.141314 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:35:45.141410 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:45.141376 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:35:45.141410 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:45.141407 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:35:46.146491 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:46.146457 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" event={"ID":"89f4bccb-46f7-4945-890c-a64eb25a073f","Type":"ContainerStarted","Data":"1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed"} Jul 22 15:35:46.436448 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:46.436340 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:46.437727 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:46.437692 2580 patch_prober.go:28] interesting pod/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 22 15:35:46.437891 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:46.437772 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 22 15:35:47.076367 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:47.076338 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:47.435833 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:47.435738 2580 patch_prober.go:28] interesting pod/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 22 15:35:47.436322 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:47.436204 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 22 15:35:48.435612 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:48.435566 2580 patch_prober.go:28] interesting pod/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 22 15:35:48.435895 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:48.435856 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 22 15:35:49.438612 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:49.438561 2580 patch_prober.go:28] interesting pod/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 22 15:35:49.439105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:49.438656 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 22 15:35:50.439220 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:50.439191 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:50.440275 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:50.440257 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:35:53.665756 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:35:53.665723 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:36:16.577754 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:16.577670 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:36:20.330436 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:20.330403 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl"] Jul 22 15:36:20.332630 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:20.332615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl" Jul 22 15:36:20.335537 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:20.335517 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b79tk\"" Jul 22 15:36:20.335655 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:20.335544 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 15:36:20.345134 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:20.342403 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl"] Jul 22 15:36:20.408061 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:20.408027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl\" (UID: \"b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl" Jul 22 15:36:20.508630 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:20.508601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl\" (UID: \"b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl" Jul 22 15:36:20.509172 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:20.509153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl\" (UID: \"b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl" Jul 22 15:36:20.644691 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:20.644609 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl" Jul 22 15:36:20.769027 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:20.769001 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl"] Jul 22 15:36:20.771705 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:36:20.771680 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ebd6ff_cb56_4a7e_9205_8bb3f5a1e186.slice/crio-0d8b8e126b6cc8b93ff5470bef61e9d742cfe20fb53bb72f4f5b2c4fe9985e4c WatchSource:0}: Error finding container 0d8b8e126b6cc8b93ff5470bef61e9d742cfe20fb53bb72f4f5b2c4fe9985e4c: Status 404 returned error can't find the container with id 0d8b8e126b6cc8b93ff5470bef61e9d742cfe20fb53bb72f4f5b2c4fe9985e4c Jul 22 15:36:21.309465 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:21.309432 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl" event={"ID":"b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186","Type":"ContainerStarted","Data":"128f5f7c8e7329332387aa3a2b2f9b90ed1e1cf4e2477241f407c3f2aa3afcc7"} Jul 22 15:36:21.309627 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:21.309484 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl" event={"ID":"b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186","Type":"ContainerStarted","Data":"0d8b8e126b6cc8b93ff5470bef61e9d742cfe20fb53bb72f4f5b2c4fe9985e4c"} Jul 22 15:36:21.327847 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:21.327791 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl" podStartSLOduration=1.327774969 podStartE2EDuration="1.327774969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:36:21.326762201 +0000 UTC m=+1906.825661979" watchObservedRunningTime="2026-07-22 15:36:21.327774969 +0000 UTC m=+1906.826674745" Jul 22 15:36:22.317132 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:22.317080 2580 generic.go:356] "Generic (PLEG): container finished" podID="b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186" containerID="128f5f7c8e7329332387aa3a2b2f9b90ed1e1cf4e2477241f407c3f2aa3afcc7" exitCode=0 Jul 22 15:36:22.317607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:22.317152 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl" event={"ID":"b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186","Type":"ContainerDied","Data":"128f5f7c8e7329332387aa3a2b2f9b90ed1e1cf4e2477241f407c3f2aa3afcc7"} Jul 22 15:36:23.453543 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:23.453520 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl" Jul 22 15:36:23.530520 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:23.530487 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186-maas-api-service-ca\") pod \"b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186\" (UID: \"b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186\") " Jul 22 15:36:23.530840 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:23.530817 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186" (UID: "b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:36:23.631753 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:23.631668 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186-maas-api-service-ca\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:36:24.327326 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:24.327301 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl" Jul 22 15:36:24.327498 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:24.327302 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl" event={"ID":"b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186","Type":"ContainerDied","Data":"0d8b8e126b6cc8b93ff5470bef61e9d742cfe20fb53bb72f4f5b2c4fe9985e4c"} Jul 22 15:36:24.327498 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:24.327418 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d8b8e126b6cc8b93ff5470bef61e9d742cfe20fb53bb72f4f5b2c4fe9985e4c" Jul 22 15:36:34.288524 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:34.288495 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:36:40.419185 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:40.419151 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl"] Jul 22 15:36:40.421775 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:40.421743 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca366ee0-mnmxl"] Jul 22 15:36:41.059324 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:41.059291 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186" path="/var/lib/kubelet/pods/b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186/volumes" Jul 22 15:36:42.972111 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:36:42.972077 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:37:31.121273 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:31.121179 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7"] Jul 22 15:37:31.121809 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:31.121446 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" containerID="cri-o://1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed" gracePeriod=30 Jul 22 15:37:34.975926 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:34.975885 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:37:35.441463 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:35.441426 2580 patch_prober.go:28] interesting pod/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 15:37:35.441616 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:35.441504 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 15:37:36.376078 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.376053 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:37:36.414639 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.414554 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-token\") pod \"89f4bccb-46f7-4945-890c-a64eb25a073f\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " Jul 22 15:37:36.414639 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.414610 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-envoy\") pod \"89f4bccb-46f7-4945-890c-a64eb25a073f\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " Jul 22 15:37:36.414860 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.414641 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-698pp\" (UniqueName: \"kubernetes.io/projected/89f4bccb-46f7-4945-890c-a64eb25a073f-kube-api-access-698pp\") pod \"89f4bccb-46f7-4945-890c-a64eb25a073f\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " Jul 22 15:37:36.414860 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.414698 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-podinfo\") pod \"89f4bccb-46f7-4945-890c-a64eb25a073f\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " Jul 22 15:37:36.414860 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.414725 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-credential-socket\") pod \"89f4bccb-46f7-4945-890c-a64eb25a073f\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " Jul 22 15:37:36.414860 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.414751 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-workload-certs\") pod \"89f4bccb-46f7-4945-890c-a64eb25a073f\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " Jul 22 15:37:36.414860 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.414817 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f4bccb-46f7-4945-890c-a64eb25a073f-istiod-ca-cert\") pod \"89f4bccb-46f7-4945-890c-a64eb25a073f\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " Jul 22 15:37:36.414860 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.414854 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-data\") pod \"89f4bccb-46f7-4945-890c-a64eb25a073f\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " Jul 22 15:37:36.415223 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.414880 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-workload-socket\") pod \"89f4bccb-46f7-4945-890c-a64eb25a073f\" (UID: \"89f4bccb-46f7-4945-890c-a64eb25a073f\") " Jul 22 15:37:36.415223 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.415097 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "89f4bccb-46f7-4945-890c-a64eb25a073f" (UID: "89f4bccb-46f7-4945-890c-a64eb25a073f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:37:36.415352 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.415330 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "89f4bccb-46f7-4945-890c-a64eb25a073f" (UID: "89f4bccb-46f7-4945-890c-a64eb25a073f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:37:36.415462 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.415440 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f4bccb-46f7-4945-890c-a64eb25a073f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "89f4bccb-46f7-4945-890c-a64eb25a073f" (UID: "89f4bccb-46f7-4945-890c-a64eb25a073f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:37:36.415835 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.415796 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "89f4bccb-46f7-4945-890c-a64eb25a073f" (UID: "89f4bccb-46f7-4945-890c-a64eb25a073f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:37:36.416189 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.416161 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-data" (OuterVolumeSpecName: "istio-data") pod "89f4bccb-46f7-4945-890c-a64eb25a073f" (UID: "89f4bccb-46f7-4945-890c-a64eb25a073f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:37:36.416981 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.416956 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-token" (OuterVolumeSpecName: "istio-token") pod "89f4bccb-46f7-4945-890c-a64eb25a073f" (UID: "89f4bccb-46f7-4945-890c-a64eb25a073f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:37:36.417069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.417052 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f4bccb-46f7-4945-890c-a64eb25a073f-kube-api-access-698pp" (OuterVolumeSpecName: "kube-api-access-698pp") pod "89f4bccb-46f7-4945-890c-a64eb25a073f" (UID: "89f4bccb-46f7-4945-890c-a64eb25a073f"). InnerVolumeSpecName "kube-api-access-698pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:37:36.417177 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.417154 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "89f4bccb-46f7-4945-890c-a64eb25a073f" (UID: "89f4bccb-46f7-4945-890c-a64eb25a073f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:37:36.417724 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.417698 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "89f4bccb-46f7-4945-890c-a64eb25a073f" (UID: "89f4bccb-46f7-4945-890c-a64eb25a073f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 15:37:36.515714 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.515686 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-podinfo\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:37:36.515714 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.515712 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-credential-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:37:36.515887 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.515726 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-workload-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:37:36.515887 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.515735 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f4bccb-46f7-4945-890c-a64eb25a073f-istiod-ca-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:37:36.515887 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.515743 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-data\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:37:36.515887 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.515750 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-workload-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:37:36.515887 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.515757 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:37:36.515887 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.515765 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f4bccb-46f7-4945-890c-a64eb25a073f-istio-envoy\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:37:36.515887 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.515772 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-698pp\" (UniqueName: \"kubernetes.io/projected/89f4bccb-46f7-4945-890c-a64eb25a073f-kube-api-access-698pp\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:37:36.648805 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.648773 2580 generic.go:356] "Generic (PLEG): container finished" podID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerID="1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed" exitCode=0 Jul 22 15:37:36.649024 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.648825 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" Jul 22 15:37:36.649024 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.648855 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" event={"ID":"89f4bccb-46f7-4945-890c-a64eb25a073f","Type":"ContainerDied","Data":"1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed"} Jul 22 15:37:36.649024 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.648946 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7" event={"ID":"89f4bccb-46f7-4945-890c-a64eb25a073f","Type":"ContainerDied","Data":"fcdc8b68bcd4982e4e74fcf500fb1cfe6da018c7f63ce73f794136a36034e9e6"} Jul 22 15:37:36.649024 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.648975 2580 scope.go:117] "RemoveContainer" containerID="1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed" Jul 22 15:37:36.657839 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.657824 2580 scope.go:117] "RemoveContainer" containerID="849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7" Jul 22 15:37:36.665387 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.665368 2580 scope.go:117] "RemoveContainer" containerID="1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed" Jul 22 15:37:36.665631 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:37:36.665605 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed\": container with ID starting with 1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed not found: ID does not exist" containerID="1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed" Jul 22 15:37:36.665704 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.665632 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed"} err="failed to get container status \"1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed\": rpc error: code = NotFound desc = could not find container \"1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed\": container with ID starting with 1b3b2c436b09e90f69b885931e4887c6bf505fe7434e41ecbb6a1c3290eba1ed not found: ID does not exist" Jul 22 15:37:36.665704 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.665649 2580 scope.go:117] "RemoveContainer" containerID="849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7" Jul 22 15:37:36.665889 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:37:36.665871 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7\": container with ID starting with 849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7 not found: ID does not exist" containerID="849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7" Jul 22 15:37:36.665945 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.665898 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7"} err="failed to get container status \"849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7\": rpc error: code = NotFound desc = could not find container \"849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7\": container with ID starting with 849fa363a9f6f65feb1d5f486aa3dbe1fa459043e0bdd6060c58e22792e124d7 not found: ID does not exist" Jul 22 15:37:36.672374 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.672355 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7"] Jul 22 15:37:36.675830 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:36.675809 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ca366ee0-openshift-default-bcc484d55-n98g7"] Jul 22 15:37:37.053546 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:37.053470 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" path="/var/lib/kubelet/pods/89f4bccb-46f7-4945-890c-a64eb25a073f/volumes" Jul 22 15:37:45.285599 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:37:45.285569 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:38:29.590373 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.590336 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v"] Jul 22 15:38:29.590944 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.590643 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" Jul 22 15:38:29.590944 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.590654 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" Jul 22 15:38:29.590944 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.590672 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" Jul 22 15:38:29.590944 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.590677 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" Jul 22 15:38:29.590944 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.590685 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186" containerName="revoke-keys" Jul 22 15:38:29.590944 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.590691 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186" containerName="revoke-keys" Jul 22 15:38:29.590944 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.590762 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" Jul 22 15:38:29.590944 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.590773 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8ebd6ff-cb56-4a7e-9205-8bb3f5a1e186" containerName="revoke-keys" Jul 22 15:38:29.593538 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.593520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.597897 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.597875 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-dd89dd-openshift-default-dockercfg-2sfh7\"" Jul 22 15:38:29.609110 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.609088 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v"] Jul 22 15:38:29.744184 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.744148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-workload-socket\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.744377 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.744236 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-podinfo\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.744377 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.744271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-envoy\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.744377 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.744354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-credential-socket\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.744546 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.744385 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-workload-certs\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.744546 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.744413 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vtwq\" (UniqueName: \"kubernetes.io/projected/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-kube-api-access-5vtwq\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.744546 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.744466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-data\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.744546 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.744514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istiod-ca-cert\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.744750 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.744572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-token\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.845423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.845344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-token\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.845423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.845382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-workload-socket\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.845423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.845403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-podinfo\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.845678 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.845432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-envoy\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.845678 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.845466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-credential-socket\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.845678 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.845499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-workload-certs\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.845678 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.845524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vtwq\" (UniqueName: \"kubernetes.io/projected/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-kube-api-access-5vtwq\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.845678 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.845569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-data\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.845678 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.845610 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istiod-ca-cert\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.845979 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.845874 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-workload-socket\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.845979 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.845944 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-data\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.846090 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.846018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-credential-socket\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.846188 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.846103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-workload-certs\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.846306 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.846290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istiod-ca-cert\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.847762 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.847734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-envoy\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.847952 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.847938 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-podinfo\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.856645 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.856621 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-token\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.861766 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.861743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vtwq\" (UniqueName: \"kubernetes.io/projected/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-kube-api-access-5vtwq\") pod \"e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:29.903433 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:29.903410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:30.043465 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:30.043440 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v"] Jul 22 15:38:30.045335 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:38:30.045311 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2e81e1_f8da_41f5_8eef_17a535b1af6b.slice/crio-526bc01de60ac82e8ad4638ba31b02250c8afe6014eccd6e65a7ba7e0ea775e7 WatchSource:0}: Error finding container 526bc01de60ac82e8ad4638ba31b02250c8afe6014eccd6e65a7ba7e0ea775e7: Status 404 returned error can't find the container with id 526bc01de60ac82e8ad4638ba31b02250c8afe6014eccd6e65a7ba7e0ea775e7 Jul 22 15:38:30.047280 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:30.047243 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:38:30.047346 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:30.047317 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:38:30.047388 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:30.047346 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:38:30.901016 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:30.900987 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" event={"ID":"2d2e81e1-f8da-41f5-8eef-17a535b1af6b","Type":"ContainerStarted","Data":"b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e"} Jul 22 15:38:30.901387 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:30.901033 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" event={"ID":"2d2e81e1-f8da-41f5-8eef-17a535b1af6b","Type":"ContainerStarted","Data":"526bc01de60ac82e8ad4638ba31b02250c8afe6014eccd6e65a7ba7e0ea775e7"} Jul 22 15:38:30.904328 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:30.904311 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:30.931769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:30.931722 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" podStartSLOduration=1.9317092150000001 podStartE2EDuration="1.931709215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:38:30.929866322 +0000 UTC m=+2036.428766098" watchObservedRunningTime="2026-07-22 15:38:30.931709215 +0000 UTC m=+2036.430608991" Jul 22 15:38:31.908831 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:31.908800 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:32.911463 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:32.911428 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:32.912514 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:32.912477 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:35.121440 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:35.121406 2580 scope.go:117] "RemoveContainer" containerID="7d5a7ae721cdbdf17e2a6560e8bba7553fa981e1885d6109f8597433ec818c67" Jul 22 15:38:37.585040 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:37.584998 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:38:38.159413 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.159383 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z"] Jul 22 15:38:38.159768 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.159754 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f4bccb-46f7-4945-890c-a64eb25a073f" containerName="istio-proxy" Jul 22 15:38:38.163994 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.163971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.167054 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.167033 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9b66ab-openshift-default-dockercfg-6hdb5\"" Jul 22 15:38:38.176142 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.176104 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z"] Jul 22 15:38:38.313050 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.313000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efff305a-0d05-4c23-807a-eff8afa85404-istiod-ca-cert\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.313258 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.313078 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efff305a-0d05-4c23-807a-eff8afa85404-istio-token\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.313258 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.313173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-credential-socket\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.313258 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.313196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-istio-envoy\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.313258 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.313214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-workload-certs\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.313258 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.313233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-istio-data\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.313258 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.313250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efff305a-0d05-4c23-807a-eff8afa85404-istio-podinfo\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.313644 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.313267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgn5c\" (UniqueName: \"kubernetes.io/projected/efff305a-0d05-4c23-807a-eff8afa85404-kube-api-access-dgn5c\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.313644 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.313291 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-workload-socket\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414105 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-workload-socket\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414288 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efff305a-0d05-4c23-807a-eff8afa85404-istiod-ca-cert\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414288 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efff305a-0d05-4c23-807a-eff8afa85404-istio-token\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414288 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-credential-socket\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414288 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-istio-envoy\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414504 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-workload-certs\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414504 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-istio-data\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414504 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efff305a-0d05-4c23-807a-eff8afa85404-istio-podinfo\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414504 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgn5c\" (UniqueName: \"kubernetes.io/projected/efff305a-0d05-4c23-807a-eff8afa85404-kube-api-access-dgn5c\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414504 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414453 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-workload-socket\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414793 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414699 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-workload-certs\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414859 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414793 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efff305a-0d05-4c23-807a-eff8afa85404-istiod-ca-cert\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414859 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414846 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-credential-socket\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.414977 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.414956 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-istio-data\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.417166 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.417139 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-istio-envoy\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.417310 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.417208 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efff305a-0d05-4c23-807a-eff8afa85404-istio-podinfo\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.426742 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.426715 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efff305a-0d05-4c23-807a-eff8afa85404-istio-token\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.426868 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.426722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgn5c\" (UniqueName: \"kubernetes.io/projected/efff305a-0d05-4c23-807a-eff8afa85404-kube-api-access-dgn5c\") pod \"e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.477169 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.477136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:38.678553 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.678527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z"] Jul 22 15:38:38.680752 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:38:38.680729 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefff305a_0d05_4c23_807a_eff8afa85404.slice/crio-51eb77e31f239bb85e42cc107d6ce9d9f762e99cd78bcfe5d32c8c3c747c03aa WatchSource:0}: Error finding container 51eb77e31f239bb85e42cc107d6ce9d9f762e99cd78bcfe5d32c8c3c747c03aa: Status 404 returned error can't find the container with id 51eb77e31f239bb85e42cc107d6ce9d9f762e99cd78bcfe5d32c8c3c747c03aa Jul 22 15:38:38.682897 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.682862 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:38:38.683001 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.682939 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:38:38.683001 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.682987 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:38:38.947312 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.946615 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" event={"ID":"efff305a-0d05-4c23-807a-eff8afa85404","Type":"ContainerStarted","Data":"2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9"} Jul 22 15:38:38.947312 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.946679 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" event={"ID":"efff305a-0d05-4c23-807a-eff8afa85404","Type":"ContainerStarted","Data":"51eb77e31f239bb85e42cc107d6ce9d9f762e99cd78bcfe5d32c8c3c747c03aa"} Jul 22 15:38:38.975180 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:38.974756 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" podStartSLOduration=0.974736344 podStartE2EDuration="974.736344ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:38:38.974505153 +0000 UTC m=+2044.473404930" watchObservedRunningTime="2026-07-22 15:38:38.974736344 +0000 UTC m=+2044.473636120" Jul 22 15:38:39.478196 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:39.478164 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:39.482772 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:39.482749 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:39.951401 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:39.951368 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:39.952424 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:39.952403 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:45.982474 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:45.982444 2580 generic.go:356] "Generic (PLEG): container finished" podID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerID="b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e" exitCode=0 Jul 22 15:38:45.982948 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:45.982522 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" event={"ID":"2d2e81e1-f8da-41f5-8eef-17a535b1af6b","Type":"ContainerDied","Data":"b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e"} Jul 22 15:38:45.982948 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:45.982871 2580 scope.go:117] "RemoveContainer" containerID="b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e" Jul 22 15:38:45.983899 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:45.983871 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:38:45.983969 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:45.983938 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:38:45.984012 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:45.983990 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:38:46.989339 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:46.989308 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" event={"ID":"2d2e81e1-f8da-41f5-8eef-17a535b1af6b","Type":"ContainerStarted","Data":"d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c"} Jul 22 15:38:47.903747 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:47.903704 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:47.904923 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:47.904900 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 15:38:47.905046 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:47.904958 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 15:38:47.912040 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:47.912007 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:48.391443 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:48.391410 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:38:48.904434 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:48.904399 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 15:38:48.904611 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:48.904452 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 15:38:49.904457 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:49.904425 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 15:38:49.904839 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:49.904486 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 15:38:50.904591 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:50.904548 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 15:38:50.904978 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:50.904609 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 15:38:51.907557 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:51.907524 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:51.908395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:51.908377 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:38:55.994269 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:55.994228 2580 patch_prober.go:28] interesting pod/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 15:38:55.994641 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:55.994279 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" podUID="efff305a-0d05-4c23-807a-eff8afa85404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 15:38:57.039980 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:57.039899 2580 generic.go:356] "Generic (PLEG): container finished" podID="efff305a-0d05-4c23-807a-eff8afa85404" containerID="2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9" exitCode=0 Jul 22 15:38:57.040503 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:57.039979 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" event={"ID":"efff305a-0d05-4c23-807a-eff8afa85404","Type":"ContainerDied","Data":"2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9"} Jul 22 15:38:57.040503 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:57.040471 2580 scope.go:117] "RemoveContainer" containerID="2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9" Jul 22 15:38:57.041446 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:57.041417 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:38:57.041555 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:57.041482 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:38:57.041555 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:57.041519 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:38:58.047158 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:58.047109 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" event={"ID":"efff305a-0d05-4c23-807a-eff8afa85404","Type":"ContainerStarted","Data":"ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985"} Jul 22 15:38:58.478133 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:58.478076 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:38:58.478362 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:58.478338 2580 patch_prober.go:28] interesting pod/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 22 15:38:58.478414 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:58.478388 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" podUID="efff305a-0d05-4c23-807a-eff8afa85404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 22 15:38:59.477566 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:59.477532 2580 patch_prober.go:28] interesting pod/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 22 15:38:59.477953 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:38:59.477587 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" podUID="efff305a-0d05-4c23-807a-eff8afa85404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 22 15:39:00.477844 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:39:00.477809 2580 patch_prober.go:28] interesting pod/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 22 15:39:00.478279 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:39:00.477860 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" podUID="efff305a-0d05-4c23-807a-eff8afa85404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 22 15:39:01.477961 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:39:01.477924 2580 patch_prober.go:28] interesting pod/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 22 15:39:01.478353 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:39:01.477977 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" podUID="efff305a-0d05-4c23-807a-eff8afa85404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 22 15:39:01.579631 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:39:01.579583 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:39:02.480718 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:39:02.480688 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:39:02.481103 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:39:02.480947 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:39:02.481672 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:39:02.481654 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:39:21.965253 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:39:21.965224 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:40:11.520498 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:11.520456 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z"] Jul 22 15:40:11.522730 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:11.522715 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z" Jul 22 15:40:11.525706 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:11.525684 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b79tk\"" Jul 22 15:40:11.525834 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:11.525723 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 15:40:11.533885 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:11.533863 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z"] Jul 22 15:40:11.577844 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:11.577816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48bc3d3-f02c-43eb-97fc-f4dc86f36e33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z\" (UID: \"f48bc3d3-f02c-43eb-97fc-f4dc86f36e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z" Jul 22 15:40:11.678310 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:11.678280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48bc3d3-f02c-43eb-97fc-f4dc86f36e33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z\" (UID: \"f48bc3d3-f02c-43eb-97fc-f4dc86f36e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z" Jul 22 15:40:11.678865 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:11.678849 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48bc3d3-f02c-43eb-97fc-f4dc86f36e33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z\" (UID: \"f48bc3d3-f02c-43eb-97fc-f4dc86f36e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z" Jul 22 15:40:11.833161 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:11.833107 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z" Jul 22 15:40:11.963072 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:11.962906 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z"] Jul 22 15:40:11.965845 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:40:11.965819 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48bc3d3_f02c_43eb_97fc_f4dc86f36e33.slice/crio-8c61e0db2dcc3b9dcf47d66b0837abbcf420e6e5bfbd705f0d29a9e91fe3133a WatchSource:0}: Error finding container 8c61e0db2dcc3b9dcf47d66b0837abbcf420e6e5bfbd705f0d29a9e91fe3133a: Status 404 returned error can't find the container with id 8c61e0db2dcc3b9dcf47d66b0837abbcf420e6e5bfbd705f0d29a9e91fe3133a Jul 22 15:40:12.379407 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:12.379326 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z" event={"ID":"f48bc3d3-f02c-43eb-97fc-f4dc86f36e33","Type":"ContainerStarted","Data":"932c7d73cea53e24d2eafd7aa161f190d3ee52d88a54be0f91ba5f2c8e0f3cb5"} Jul 22 15:40:12.379407 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:12.379373 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z" event={"ID":"f48bc3d3-f02c-43eb-97fc-f4dc86f36e33","Type":"ContainerStarted","Data":"8c61e0db2dcc3b9dcf47d66b0837abbcf420e6e5bfbd705f0d29a9e91fe3133a"} Jul 22 15:40:12.396245 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:12.396194 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z" podStartSLOduration=1.39618054 podStartE2EDuration="1.39618054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:40:12.394488579 +0000 UTC m=+2137.893388346" watchObservedRunningTime="2026-07-22 15:40:12.39618054 +0000 UTC m=+2137.895080313" Jul 22 15:40:13.386039 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:13.386004 2580 generic.go:356] "Generic (PLEG): container finished" podID="f48bc3d3-f02c-43eb-97fc-f4dc86f36e33" containerID="932c7d73cea53e24d2eafd7aa161f190d3ee52d88a54be0f91ba5f2c8e0f3cb5" exitCode=0 Jul 22 15:40:13.386423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:13.386060 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z" event={"ID":"f48bc3d3-f02c-43eb-97fc-f4dc86f36e33","Type":"ContainerDied","Data":"932c7d73cea53e24d2eafd7aa161f190d3ee52d88a54be0f91ba5f2c8e0f3cb5"} Jul 22 15:40:14.525145 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:14.525103 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z" Jul 22 15:40:14.605923 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:14.605894 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48bc3d3-f02c-43eb-97fc-f4dc86f36e33-maas-api-service-ca\") pod \"f48bc3d3-f02c-43eb-97fc-f4dc86f36e33\" (UID: \"f48bc3d3-f02c-43eb-97fc-f4dc86f36e33\") " Jul 22 15:40:14.606271 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:14.606250 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48bc3d3-f02c-43eb-97fc-f4dc86f36e33-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f48bc3d3-f02c-43eb-97fc-f4dc86f36e33" (UID: "f48bc3d3-f02c-43eb-97fc-f4dc86f36e33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:40:14.706968 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:14.706881 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48bc3d3-f02c-43eb-97fc-f4dc86f36e33-maas-api-service-ca\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:40:15.170584 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:15.170553 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:40:15.401037 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:15.400960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z" event={"ID":"f48bc3d3-f02c-43eb-97fc-f4dc86f36e33","Type":"ContainerDied","Data":"8c61e0db2dcc3b9dcf47d66b0837abbcf420e6e5bfbd705f0d29a9e91fe3133a"} Jul 22 15:40:15.401037 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:15.400998 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z" Jul 22 15:40:15.401037 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:15.401007 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c61e0db2dcc3b9dcf47d66b0837abbcf420e6e5bfbd705f0d29a9e91fe3133a" Jul 22 15:40:26.637771 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:26.637739 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z"] Jul 22 15:40:26.641603 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:26.641581 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dd89dd-qpj8z"] Jul 22 15:40:27.054293 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:27.054221 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f48bc3d3-f02c-43eb-97fc-f4dc86f36e33" path="/var/lib/kubelet/pods/f48bc3d3-f02c-43eb-97fc-f4dc86f36e33/volumes" Jul 22 15:40:27.570785 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:27.570747 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:40:33.179634 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:33.179603 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v"] Jul 22 15:40:33.180073 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:33.179868 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerName="istio-proxy" containerID="cri-o://d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c" gracePeriod=30 Jul 22 15:40:36.910174 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:36.910109 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 15:40:36.910544 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:36.910209 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 15:40:38.432330 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.432303 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:40:38.512357 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.512277 2580 generic.go:356] "Generic (PLEG): container finished" podID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerID="d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c" exitCode=0 Jul 22 15:40:38.512357 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.512321 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" event={"ID":"2d2e81e1-f8da-41f5-8eef-17a535b1af6b","Type":"ContainerDied","Data":"d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c"} Jul 22 15:40:38.512357 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.512329 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" Jul 22 15:40:38.512562 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.512376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v" event={"ID":"2d2e81e1-f8da-41f5-8eef-17a535b1af6b","Type":"ContainerDied","Data":"526bc01de60ac82e8ad4638ba31b02250c8afe6014eccd6e65a7ba7e0ea775e7"} Jul 22 15:40:38.512562 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.512406 2580 scope.go:117] "RemoveContainer" containerID="d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c" Jul 22 15:40:38.520823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.520802 2580 scope.go:117] "RemoveContainer" containerID="b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e" Jul 22 15:40:38.530408 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.530387 2580 scope.go:117] "RemoveContainer" containerID="d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c" Jul 22 15:40:38.530671 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:40:38.530649 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c\": container with ID starting with d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c not found: ID does not exist" containerID="d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c" Jul 22 15:40:38.530726 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.530678 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c"} err="failed to get container status \"d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c\": rpc error: code = NotFound desc = could not find container \"d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c\": container with ID starting with d2f20139b6272e2e364053d733eb834e6251fd23fd72ed3bb2aa2d7eb14e116c not found: ID does not exist" Jul 22 15:40:38.530726 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.530700 2580 scope.go:117] "RemoveContainer" containerID="b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e" Jul 22 15:40:38.530935 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:40:38.530920 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e\": container with ID starting with b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e not found: ID does not exist" containerID="b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e" Jul 22 15:40:38.530975 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.530940 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e"} err="failed to get container status \"b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e\": rpc error: code = NotFound desc = could not find container \"b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e\": container with ID starting with b00226b9fbf9807a54e715730db9ccb895a87c1968f5aa2e19c537283741760e not found: ID does not exist" Jul 22 15:40:38.605812 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.605790 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vtwq\" (UniqueName: \"kubernetes.io/projected/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-kube-api-access-5vtwq\") pod \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " Jul 22 15:40:38.605908 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.605826 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-workload-socket\") pod \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " Jul 22 15:40:38.605908 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.605842 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-token\") pod \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " Jul 22 15:40:38.606028 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606012 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-podinfo\") pod \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " Jul 22 15:40:38.606085 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606054 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d2e81e1-f8da-41f5-8eef-17a535b1af6b" (UID: "2d2e81e1-f8da-41f5-8eef-17a535b1af6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:40:38.606085 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606066 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-credential-socket\") pod \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " Jul 22 15:40:38.606224 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606094 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-workload-certs\") pod \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " Jul 22 15:40:38.606224 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606151 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-envoy\") pod \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " Jul 22 15:40:38.606224 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606202 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istiod-ca-cert\") pod \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " Jul 22 15:40:38.606382 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606242 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-data\") pod \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\" (UID: \"2d2e81e1-f8da-41f5-8eef-17a535b1af6b\") " Jul 22 15:40:38.606382 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606314 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d2e81e1-f8da-41f5-8eef-17a535b1af6b" (UID: "2d2e81e1-f8da-41f5-8eef-17a535b1af6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:40:38.606382 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606359 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d2e81e1-f8da-41f5-8eef-17a535b1af6b" (UID: "2d2e81e1-f8da-41f5-8eef-17a535b1af6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:40:38.606629 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606568 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-workload-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:40:38.606629 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606592 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-credential-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:40:38.606629 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606607 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-workload-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:40:38.606629 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.606627 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d2e81e1-f8da-41f5-8eef-17a535b1af6b" (UID: "2d2e81e1-f8da-41f5-8eef-17a535b1af6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:40:38.607085 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.607066 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d2e81e1-f8da-41f5-8eef-17a535b1af6b" (UID: "2d2e81e1-f8da-41f5-8eef-17a535b1af6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:40:38.608415 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.608397 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d2e81e1-f8da-41f5-8eef-17a535b1af6b" (UID: "2d2e81e1-f8da-41f5-8eef-17a535b1af6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:40:38.608519 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.608410 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-kube-api-access-5vtwq" (OuterVolumeSpecName: "kube-api-access-5vtwq") pod "2d2e81e1-f8da-41f5-8eef-17a535b1af6b" (UID: "2d2e81e1-f8da-41f5-8eef-17a535b1af6b"). InnerVolumeSpecName "kube-api-access-5vtwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:40:38.608519 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.608501 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d2e81e1-f8da-41f5-8eef-17a535b1af6b" (UID: "2d2e81e1-f8da-41f5-8eef-17a535b1af6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 15:40:38.608631 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.608525 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d2e81e1-f8da-41f5-8eef-17a535b1af6b" (UID: "2d2e81e1-f8da-41f5-8eef-17a535b1af6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:40:38.707675 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.707635 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-envoy\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:40:38.707675 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.707673 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istiod-ca-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:40:38.707675 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.707682 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-data\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:40:38.707843 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.707690 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vtwq\" (UniqueName: \"kubernetes.io/projected/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-kube-api-access-5vtwq\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:40:38.707843 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.707699 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:40:38.707843 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.707709 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2e81e1-f8da-41f5-8eef-17a535b1af6b-istio-podinfo\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:40:38.836665 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.836640 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v"] Jul 22 15:40:38.840150 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:38.840103 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-dd89dd-openshift-default-755b5dd59-v8p8v"] Jul 22 15:40:39.054242 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:39.054218 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" path="/var/lib/kubelet/pods/2d2e81e1-f8da-41f5-8eef-17a535b1af6b/volumes" Jul 22 15:40:47.077967 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:47.077934 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:40:55.778214 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:55.778183 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z"] Jul 22 15:40:55.778601 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:55.778452 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" podUID="efff305a-0d05-4c23-807a-eff8afa85404" containerName="istio-proxy" containerID="cri-o://ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985" gracePeriod=30 Jul 22 15:40:56.726993 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.726953 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d"] Jul 22 15:40:56.727371 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.727354 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerName="istio-proxy" Jul 22 15:40:56.727463 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.727374 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerName="istio-proxy" Jul 22 15:40:56.727463 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.727392 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerName="istio-proxy" Jul 22 15:40:56.727463 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.727400 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerName="istio-proxy" Jul 22 15:40:56.727463 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.727438 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f48bc3d3-f02c-43eb-97fc-f4dc86f36e33" containerName="revoke-keys" Jul 22 15:40:56.727463 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.727447 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f48bc3d3-f02c-43eb-97fc-f4dc86f36e33" containerName="revoke-keys" Jul 22 15:40:56.727719 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.727522 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerName="istio-proxy" Jul 22 15:40:56.727719 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.727532 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d2e81e1-f8da-41f5-8eef-17a535b1af6b" containerName="istio-proxy" Jul 22 15:40:56.727719 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.727547 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f48bc3d3-f02c-43eb-97fc-f4dc86f36e33" containerName="revoke-keys" Jul 22 15:40:56.729757 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.729736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.735061 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.735044 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-81c749-openshift-default-dockercfg-6f6hz\"" Jul 22 15:40:56.747058 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.747033 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d"] Jul 22 15:40:56.749877 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.749855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-workload-certs\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.749969 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.749897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzqh8\" (UniqueName: \"kubernetes.io/projected/5bbe6efd-789d-4b1a-98ff-2655eecf3575-kube-api-access-nzqh8\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.749969 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.749918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-data\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.749969 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.749935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istiod-ca-cert\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.750079 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.750014 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-podinfo\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.750079 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.750048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-token\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.750079 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.750068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-envoy\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.750201 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.750088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-credential-socket\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.750201 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.750106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-workload-socket\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.850994 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.850961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-workload-certs\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.851501 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.851008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqh8\" (UniqueName: \"kubernetes.io/projected/5bbe6efd-789d-4b1a-98ff-2655eecf3575-kube-api-access-nzqh8\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.851501 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.851030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-data\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.851501 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.851046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istiod-ca-cert\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.851501 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.851070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-podinfo\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.851501 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.851100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-token\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.851501 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.851153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-envoy\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.851501 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.851187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-credential-socket\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.851501 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.851207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-workload-socket\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.851501 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.851477 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-workload-certs\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.852023 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.851731 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-data\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.852023 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.851755 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-workload-socket\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.852023 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.851816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-credential-socket\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.852215 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.852104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istiod-ca-cert\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.853551 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.853529 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-envoy\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.853754 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.853736 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-podinfo\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.859534 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.859510 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-token\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:56.859771 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:56.859752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqh8\" (UniqueName: \"kubernetes.io/projected/5bbe6efd-789d-4b1a-98ff-2655eecf3575-kube-api-access-nzqh8\") pod \"e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:57.042167 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:57.042059 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:57.183914 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:57.183888 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d"] Jul 22 15:40:57.185777 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:40:57.185743 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbe6efd_789d_4b1a_98ff_2655eecf3575.slice/crio-90a79c9b422ed45b23d0838b5f870d5750873f96d47d74c535e7068d9a66ddff WatchSource:0}: Error finding container 90a79c9b422ed45b23d0838b5f870d5750873f96d47d74c535e7068d9a66ddff: Status 404 returned error can't find the container with id 90a79c9b422ed45b23d0838b5f870d5750873f96d47d74c535e7068d9a66ddff Jul 22 15:40:57.187788 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:57.187757 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:40:57.187885 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:57.187834 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:40:57.187885 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:57.187876 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:40:57.603183 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:57.603144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" event={"ID":"5bbe6efd-789d-4b1a-98ff-2655eecf3575","Type":"ContainerStarted","Data":"da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566"} Jul 22 15:40:57.603465 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:57.603208 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" event={"ID":"5bbe6efd-789d-4b1a-98ff-2655eecf3575","Type":"ContainerStarted","Data":"90a79c9b422ed45b23d0838b5f870d5750873f96d47d74c535e7068d9a66ddff"} Jul 22 15:40:58.042359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:58.042283 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:58.046800 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:58.046773 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:58.073014 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:58.072971 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" podStartSLOduration=2.072957558 podStartE2EDuration="2.072957558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:40:57.62786948 +0000 UTC m=+2183.126769256" watchObservedRunningTime="2026-07-22 15:40:58.072957558 +0000 UTC m=+2183.571857325" Jul 22 15:40:58.608023 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:58.607993 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:40:58.608782 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:40:58.608766 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:41:01.022831 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.022799 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:41:01.087670 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.087637 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efff305a-0d05-4c23-807a-eff8afa85404-istiod-ca-cert\") pod \"efff305a-0d05-4c23-807a-eff8afa85404\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " Jul 22 15:41:01.087811 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.087677 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgn5c\" (UniqueName: \"kubernetes.io/projected/efff305a-0d05-4c23-807a-eff8afa85404-kube-api-access-dgn5c\") pod \"efff305a-0d05-4c23-807a-eff8afa85404\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " Jul 22 15:41:01.087811 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.087722 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efff305a-0d05-4c23-807a-eff8afa85404-istio-podinfo\") pod \"efff305a-0d05-4c23-807a-eff8afa85404\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " Jul 22 15:41:01.087811 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.087741 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-workload-certs\") pod \"efff305a-0d05-4c23-807a-eff8afa85404\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " Jul 22 15:41:01.087811 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.087756 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efff305a-0d05-4c23-807a-eff8afa85404-istio-token\") pod \"efff305a-0d05-4c23-807a-eff8afa85404\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " Jul 22 15:41:01.087811 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.087776 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-credential-socket\") pod \"efff305a-0d05-4c23-807a-eff8afa85404\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " Jul 22 15:41:01.088073 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.087816 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-istio-data\") pod \"efff305a-0d05-4c23-807a-eff8afa85404\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " Jul 22 15:41:01.088073 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.087854 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-istio-envoy\") pod \"efff305a-0d05-4c23-807a-eff8afa85404\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " Jul 22 15:41:01.088073 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.087893 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-workload-socket\") pod \"efff305a-0d05-4c23-807a-eff8afa85404\" (UID: \"efff305a-0d05-4c23-807a-eff8afa85404\") " Jul 22 15:41:01.088073 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.087997 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efff305a-0d05-4c23-807a-eff8afa85404-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "efff305a-0d05-4c23-807a-eff8afa85404" (UID: "efff305a-0d05-4c23-807a-eff8afa85404"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:41:01.088324 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.088099 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "efff305a-0d05-4c23-807a-eff8afa85404" (UID: "efff305a-0d05-4c23-807a-eff8afa85404"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:41:01.088324 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.088166 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "efff305a-0d05-4c23-807a-eff8afa85404" (UID: "efff305a-0d05-4c23-807a-eff8afa85404"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:41:01.088324 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.088310 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-workload-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:01.088487 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.088330 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-credential-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:01.088487 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.088347 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efff305a-0d05-4c23-807a-eff8afa85404-istiod-ca-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:01.088487 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.088349 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "efff305a-0d05-4c23-807a-eff8afa85404" (UID: "efff305a-0d05-4c23-807a-eff8afa85404"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:41:01.088965 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.088936 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-istio-data" (OuterVolumeSpecName: "istio-data") pod "efff305a-0d05-4c23-807a-eff8afa85404" (UID: "efff305a-0d05-4c23-807a-eff8afa85404"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:41:01.090037 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.090011 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efff305a-0d05-4c23-807a-eff8afa85404-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "efff305a-0d05-4c23-807a-eff8afa85404" (UID: "efff305a-0d05-4c23-807a-eff8afa85404"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 15:41:01.090037 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.090024 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efff305a-0d05-4c23-807a-eff8afa85404-istio-token" (OuterVolumeSpecName: "istio-token") pod "efff305a-0d05-4c23-807a-eff8afa85404" (UID: "efff305a-0d05-4c23-807a-eff8afa85404"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:41:01.090213 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.090082 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "efff305a-0d05-4c23-807a-eff8afa85404" (UID: "efff305a-0d05-4c23-807a-eff8afa85404"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:41:01.090257 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.090237 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efff305a-0d05-4c23-807a-eff8afa85404-kube-api-access-dgn5c" (OuterVolumeSpecName: "kube-api-access-dgn5c") pod "efff305a-0d05-4c23-807a-eff8afa85404" (UID: "efff305a-0d05-4c23-807a-eff8afa85404"). InnerVolumeSpecName "kube-api-access-dgn5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:41:01.188658 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.188630 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-istio-data\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:01.188658 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.188656 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-istio-envoy\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:01.188658 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.188664 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff305a-0d05-4c23-807a-eff8afa85404-workload-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:01.188855 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.188672 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgn5c\" (UniqueName: \"kubernetes.io/projected/efff305a-0d05-4c23-807a-eff8afa85404-kube-api-access-dgn5c\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:01.188855 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.188681 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efff305a-0d05-4c23-807a-eff8afa85404-istio-podinfo\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:01.188855 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.188690 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efff305a-0d05-4c23-807a-eff8afa85404-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:01.623642 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.623605 2580 generic.go:356] "Generic (PLEG): container finished" podID="efff305a-0d05-4c23-807a-eff8afa85404" containerID="ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985" exitCode=0 Jul 22 15:41:01.623844 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.623677 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" Jul 22 15:41:01.623844 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.623687 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" event={"ID":"efff305a-0d05-4c23-807a-eff8afa85404","Type":"ContainerDied","Data":"ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985"} Jul 22 15:41:01.623844 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.623747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z" event={"ID":"efff305a-0d05-4c23-807a-eff8afa85404","Type":"ContainerDied","Data":"51eb77e31f239bb85e42cc107d6ce9d9f762e99cd78bcfe5d32c8c3c747c03aa"} Jul 22 15:41:01.623844 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.623777 2580 scope.go:117] "RemoveContainer" containerID="ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985" Jul 22 15:41:01.633002 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.632983 2580 scope.go:117] "RemoveContainer" containerID="2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9" Jul 22 15:41:01.640368 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.640351 2580 scope.go:117] "RemoveContainer" containerID="ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985" Jul 22 15:41:01.640614 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:41:01.640598 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985\": container with ID starting with ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985 not found: ID does not exist" containerID="ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985" Jul 22 15:41:01.640665 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.640621 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985"} err="failed to get container status \"ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985\": rpc error: code = NotFound desc = could not find container \"ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985\": container with ID starting with ff067f86cea0dbd79a6c4eca055d0443346476d91fa5f0cb69c074fc5794e985 not found: ID does not exist" Jul 22 15:41:01.640665 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.640637 2580 scope.go:117] "RemoveContainer" containerID="2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9" Jul 22 15:41:01.640852 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:41:01.640835 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9\": container with ID starting with 2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9 not found: ID does not exist" containerID="2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9" Jul 22 15:41:01.640894 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.640859 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9"} err="failed to get container status \"2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9\": rpc error: code = NotFound desc = could not find container \"2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9\": container with ID starting with 2dac8eb5d3a30407cb408245d6defa90c63850db14f34f3ae459b7a60021e4e9 not found: ID does not exist" Jul 22 15:41:01.644979 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.644959 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z"] Jul 22 15:41:01.649887 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:01.649867 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9b66ab-openshift-default-785fc99c9f-mz59z"] Jul 22 15:41:03.053843 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:03.053816 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efff305a-0d05-4c23-807a-eff8afa85404" path="/var/lib/kubelet/pods/efff305a-0d05-4c23-807a-eff8afa85404/volumes" Jul 22 15:41:04.996231 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:04.996203 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h"] Jul 22 15:41:04.996581 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:04.996504 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efff305a-0d05-4c23-807a-eff8afa85404" containerName="istio-proxy" Jul 22 15:41:04.996581 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:04.996515 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="efff305a-0d05-4c23-807a-eff8afa85404" containerName="istio-proxy" Jul 22 15:41:04.996581 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:04.996532 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efff305a-0d05-4c23-807a-eff8afa85404" containerName="istio-proxy" Jul 22 15:41:04.996581 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:04.996538 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="efff305a-0d05-4c23-807a-eff8afa85404" containerName="istio-proxy" Jul 22 15:41:04.996715 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:04.996601 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="efff305a-0d05-4c23-807a-eff8afa85404" containerName="istio-proxy" Jul 22 15:41:04.996715 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:04.996609 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="efff305a-0d05-4c23-807a-eff8afa85404" containerName="istio-proxy" Jul 22 15:41:04.999383 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:04.999363 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.002579 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.002558 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b47c9e-openshift-default-dockercfg-l9z8n\"" Jul 22 15:41:05.015444 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.015417 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h"] Jul 22 15:41:05.018330 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.018304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istiod-ca-cert\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.018460 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.018366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-envoy\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.018460 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.018412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-data\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.018591 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.018456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-podinfo\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.018591 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.018523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd7j9\" (UniqueName: \"kubernetes.io/projected/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-kube-api-access-qd7j9\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.018591 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.018554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-workload-socket\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.018591 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.018576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-workload-certs\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.018782 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.018689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-credential-socket\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.018782 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.018741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-token\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.119903 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.119863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd7j9\" (UniqueName: \"kubernetes.io/projected/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-kube-api-access-qd7j9\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.120073 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.119909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-workload-socket\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.120073 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.119938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-workload-certs\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.120073 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.119991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-credential-socket\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.120073 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.120031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-token\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.120352 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.120105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istiod-ca-cert\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.120352 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.120189 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-envoy\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.120352 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.120215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-data\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.120352 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.120246 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-podinfo\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.120352 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.120330 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-workload-socket\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.120795 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.120761 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istiod-ca-cert\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.120922 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.120903 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-credential-socket\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.120992 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.120936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-workload-certs\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.121063 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.121041 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-data\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.122659 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.122634 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-envoy\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.122776 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.122759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-podinfo\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.128280 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.128256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-token\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.128412 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.128387 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd7j9\" (UniqueName: \"kubernetes.io/projected/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-kube-api-access-qd7j9\") pod \"e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.309654 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.309630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:05.453071 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.453047 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h"] Jul 22 15:41:05.455339 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:41:05.455310 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2d483f_1ab2_4e1c_bbba_ae5e1c804458.slice/crio-6e3248fef16e0c17c44fbd5702f600a1e80312ad262ab793ffc64fd990e8e5c0 WatchSource:0}: Error finding container 6e3248fef16e0c17c44fbd5702f600a1e80312ad262ab793ffc64fd990e8e5c0: Status 404 returned error can't find the container with id 6e3248fef16e0c17c44fbd5702f600a1e80312ad262ab793ffc64fd990e8e5c0 Jul 22 15:41:05.457852 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.457822 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:41:05.457918 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.457882 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:41:05.457918 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.457912 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:41:05.645090 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.645003 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" event={"ID":"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458","Type":"ContainerStarted","Data":"5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7"} Jul 22 15:41:05.645090 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.645063 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" event={"ID":"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458","Type":"ContainerStarted","Data":"6e3248fef16e0c17c44fbd5702f600a1e80312ad262ab793ffc64fd990e8e5c0"} Jul 22 15:41:05.670178 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:05.670127 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" podStartSLOduration=1.670100731 podStartE2EDuration="1.670100731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:41:05.668840249 +0000 UTC m=+2191.167740051" watchObservedRunningTime="2026-07-22 15:41:05.670100731 +0000 UTC m=+2191.169000500" Jul 22 15:41:06.310109 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:06.310070 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:06.315393 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:06.315367 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:06.649475 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:06.649396 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:06.650204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:06.650185 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:07.483404 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:07.483372 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:41:14.650275 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:14.650232 2580 patch_prober.go:28] interesting pod/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 15:41:14.650724 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:14.650307 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 15:41:15.690663 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:15.690632 2580 generic.go:356] "Generic (PLEG): container finished" podID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerID="da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566" exitCode=0 Jul 22 15:41:15.691070 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:15.690707 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" event={"ID":"5bbe6efd-789d-4b1a-98ff-2655eecf3575","Type":"ContainerDied","Data":"da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566"} Jul 22 15:41:15.691152 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:15.691097 2580 scope.go:117] "RemoveContainer" containerID="da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566" Jul 22 15:41:15.692111 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:15.692083 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:41:15.692241 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:15.692172 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:41:15.692241 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:15.692213 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:41:16.698371 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:16.698337 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" event={"ID":"5bbe6efd-789d-4b1a-98ff-2655eecf3575","Type":"ContainerStarted","Data":"d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf"} Jul 22 15:41:17.042858 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:17.042766 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:41:17.043009 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:17.042962 2580 patch_prober.go:28] interesting pod/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 22 15:41:17.043051 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:17.043018 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 22 15:41:18.043174 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:18.043135 2580 patch_prober.go:28] interesting pod/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 22 15:41:18.043847 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:18.043794 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 22 15:41:19.042870 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:19.042820 2580 patch_prober.go:28] interesting pod/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 22 15:41:19.043090 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:19.042901 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 22 15:41:20.042957 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:20.042921 2580 patch_prober.go:28] interesting pod/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 22 15:41:20.043560 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:20.043528 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 22 15:41:21.046501 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:21.046467 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:41:21.046936 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:21.046815 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:41:21.047465 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:21.047444 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:41:23.975333 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:23.975295 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:41:24.749047 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:24.748969 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerID="5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7" exitCode=0 Jul 22 15:41:24.749235 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:24.749041 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" event={"ID":"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458","Type":"ContainerDied","Data":"5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7"} Jul 22 15:41:24.749474 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:24.749459 2580 scope.go:117] "RemoveContainer" containerID="5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7" Jul 22 15:41:24.750434 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:24.750411 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:41:24.750504 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:24.750478 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:41:24.750539 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:24.750508 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:41:25.755599 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:25.755561 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" event={"ID":"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458","Type":"ContainerStarted","Data":"daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc"} Jul 22 15:41:26.310397 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:26.310362 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:26.310550 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:26.310513 2580 patch_prober.go:28] interesting pod/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 22 15:41:26.310624 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:26.310567 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 22 15:41:27.310203 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:27.310173 2580 patch_prober.go:28] interesting pod/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 22 15:41:27.310665 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:27.310232 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 22 15:41:28.310417 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:28.310385 2580 patch_prober.go:28] interesting pod/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 22 15:41:28.310807 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:28.310466 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 22 15:41:29.310420 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:29.310387 2580 patch_prober.go:28] interesting pod/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 22 15:41:29.310806 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:29.310456 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 22 15:41:30.314394 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:30.314365 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:30.314853 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:30.314648 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:30.315335 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:30.315317 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:41:35.154394 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:35.154365 2580 scope.go:117] "RemoveContainer" containerID="10b996cc49d412c696ca81fcb86316b84c2c62590d0d6ee38cf256b328564f7b" Jul 22 15:41:39.572470 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:39.572437 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:41:47.236785 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:47.236749 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d"] Jul 22 15:41:47.237317 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:47.236965 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" containerID="cri-o://d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf" gracePeriod=30 Jul 22 15:41:51.049689 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:51.049646 2580 patch_prober.go:28] interesting pod/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 15:41:51.050140 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:51.049745 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 15:41:52.489005 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.488983 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:41:52.536742 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.536681 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-data\") pod \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " Jul 22 15:41:52.536742 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.536710 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-envoy\") pod \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " Jul 22 15:41:52.536742 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.536737 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-workload-certs\") pod \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " Jul 22 15:41:52.537006 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.536760 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istiod-ca-cert\") pod \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " Jul 22 15:41:52.537006 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.536801 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-credential-socket\") pod \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " Jul 22 15:41:52.537006 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.536853 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-podinfo\") pod \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " Jul 22 15:41:52.537006 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.536906 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-workload-socket\") pod \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " Jul 22 15:41:52.537006 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.536951 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzqh8\" (UniqueName: \"kubernetes.io/projected/5bbe6efd-789d-4b1a-98ff-2655eecf3575-kube-api-access-nzqh8\") pod \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " Jul 22 15:41:52.537006 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.536994 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-token\") pod \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\" (UID: \"5bbe6efd-789d-4b1a-98ff-2655eecf3575\") " Jul 22 15:41:52.537332 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.537041 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5bbe6efd-789d-4b1a-98ff-2655eecf3575" (UID: "5bbe6efd-789d-4b1a-98ff-2655eecf3575"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:41:52.537332 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.537159 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5bbe6efd-789d-4b1a-98ff-2655eecf3575" (UID: "5bbe6efd-789d-4b1a-98ff-2655eecf3575"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:41:52.537452 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.537331 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-workload-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:52.537452 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.537344 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5bbe6efd-789d-4b1a-98ff-2655eecf3575" (UID: "5bbe6efd-789d-4b1a-98ff-2655eecf3575"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:41:52.537452 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.537352 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-credential-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:52.537671 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.537656 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-data" (OuterVolumeSpecName: "istio-data") pod "5bbe6efd-789d-4b1a-98ff-2655eecf3575" (UID: "5bbe6efd-789d-4b1a-98ff-2655eecf3575"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:41:52.537731 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.537697 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5bbe6efd-789d-4b1a-98ff-2655eecf3575" (UID: "5bbe6efd-789d-4b1a-98ff-2655eecf3575"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:41:52.539113 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.539093 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-token" (OuterVolumeSpecName: "istio-token") pod "5bbe6efd-789d-4b1a-98ff-2655eecf3575" (UID: "5bbe6efd-789d-4b1a-98ff-2655eecf3575"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:41:52.539241 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.539205 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5bbe6efd-789d-4b1a-98ff-2655eecf3575" (UID: "5bbe6efd-789d-4b1a-98ff-2655eecf3575"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 15:41:52.539370 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.539349 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbe6efd-789d-4b1a-98ff-2655eecf3575-kube-api-access-nzqh8" (OuterVolumeSpecName: "kube-api-access-nzqh8") pod "5bbe6efd-789d-4b1a-98ff-2655eecf3575" (UID: "5bbe6efd-789d-4b1a-98ff-2655eecf3575"). InnerVolumeSpecName "kube-api-access-nzqh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:41:52.539415 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.539398 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5bbe6efd-789d-4b1a-98ff-2655eecf3575" (UID: "5bbe6efd-789d-4b1a-98ff-2655eecf3575"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:41:52.638473 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.638444 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzqh8\" (UniqueName: \"kubernetes.io/projected/5bbe6efd-789d-4b1a-98ff-2655eecf3575-kube-api-access-nzqh8\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:52.638473 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.638469 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:52.638473 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.638478 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-data\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:52.638679 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.638486 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-envoy\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:52.638679 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.638495 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istiod-ca-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:52.638679 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.638502 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbe6efd-789d-4b1a-98ff-2655eecf3575-istio-podinfo\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:52.638679 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.638510 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbe6efd-789d-4b1a-98ff-2655eecf3575-workload-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:41:52.881260 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.881226 2580 generic.go:356] "Generic (PLEG): container finished" podID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerID="d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf" exitCode=0 Jul 22 15:41:52.881458 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.881292 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" Jul 22 15:41:52.881458 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.881308 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" event={"ID":"5bbe6efd-789d-4b1a-98ff-2655eecf3575","Type":"ContainerDied","Data":"d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf"} Jul 22 15:41:52.881458 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.881361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d" event={"ID":"5bbe6efd-789d-4b1a-98ff-2655eecf3575","Type":"ContainerDied","Data":"90a79c9b422ed45b23d0838b5f870d5750873f96d47d74c535e7068d9a66ddff"} Jul 22 15:41:52.881458 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.881386 2580 scope.go:117] "RemoveContainer" containerID="d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf" Jul 22 15:41:52.890375 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.890360 2580 scope.go:117] "RemoveContainer" containerID="da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566" Jul 22 15:41:52.898030 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.898014 2580 scope.go:117] "RemoveContainer" containerID="d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf" Jul 22 15:41:52.898280 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:41:52.898262 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf\": container with ID starting with d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf not found: ID does not exist" containerID="d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf" Jul 22 15:41:52.898371 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.898289 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf"} err="failed to get container status \"d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf\": rpc error: code = NotFound desc = could not find container \"d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf\": container with ID starting with d8455d9159ca1537533bb74860700ab7d039dc312d3e7dddbb375a9840e218bf not found: ID does not exist" Jul 22 15:41:52.898371 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.898311 2580 scope.go:117] "RemoveContainer" containerID="da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566" Jul 22 15:41:52.898582 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:41:52.898565 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566\": container with ID starting with da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566 not found: ID does not exist" containerID="da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566" Jul 22 15:41:52.898622 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.898587 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566"} err="failed to get container status \"da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566\": rpc error: code = NotFound desc = could not find container \"da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566\": container with ID starting with da16bc9bf820c42f52ef4851f41d299e9d6b489c757033e22f31cf5319d7b566 not found: ID does not exist" Jul 22 15:41:52.903412 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.903393 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d"] Jul 22 15:41:52.909990 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:52.909971 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-81c749-openshift-default-685b4f9d59-x5h5d"] Jul 22 15:41:53.053938 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:41:53.053910 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" path="/var/lib/kubelet/pods/5bbe6efd-789d-4b1a-98ff-2655eecf3575/volumes" Jul 22 15:42:08.074475 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:08.074388 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:42:14.455306 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:14.455268 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h"] Jul 22 15:42:14.455793 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:14.455539 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerName="istio-proxy" containerID="cri-o://daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc" gracePeriod=30 Jul 22 15:42:15.317707 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.317669 2580 patch_prober.go:28] interesting pod/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 22 15:42:15.317855 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.317735 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 15:42:15.337714 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.337659 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4"] Jul 22 15:42:15.338368 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.338346 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" Jul 22 15:42:15.338368 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.338368 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" Jul 22 15:42:15.338509 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.338457 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" Jul 22 15:42:15.341321 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.341303 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.345014 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.344994 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-6f1c00-openshift-default-dockercfg-xw6fd\"" Jul 22 15:42:15.352988 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.352969 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4"] Jul 22 15:42:15.434038 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.434004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-workload-certs\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.434038 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.434043 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5t8\" (UniqueName: \"kubernetes.io/projected/79d6dd09-2686-40ff-99c8-785c5a79116c-kube-api-access-sb5t8\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.434321 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.434077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-envoy\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.434321 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.434222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-credential-socket\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.434321 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.434270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-podinfo\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.434321 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.434299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d6dd09-2686-40ff-99c8-785c5a79116c-istiod-ca-cert\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.434555 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.434361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-data\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.434555 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.434402 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-token\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.434555 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.434429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-workload-socket\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.534939 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.534906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-credential-socket\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.534939 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.534943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-podinfo\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.535484 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.534971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d6dd09-2686-40ff-99c8-785c5a79116c-istiod-ca-cert\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.535484 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.535023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-data\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.535484 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.535066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-token\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.535484 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.535098 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-workload-socket\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.535484 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.535153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-workload-certs\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.535484 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.535181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5t8\" (UniqueName: \"kubernetes.io/projected/79d6dd09-2686-40ff-99c8-785c5a79116c-kube-api-access-sb5t8\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.535484 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.535226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-envoy\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.535484 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.535370 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-credential-socket\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.535484 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.535406 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-data\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.535967 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.535720 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-workload-certs\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.535967 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.535726 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-workload-socket\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.535967 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.535769 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d6dd09-2686-40ff-99c8-785c5a79116c-istiod-ca-cert\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.537456 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.537437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-podinfo\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.537617 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.537600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-envoy\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.543789 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.543769 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-token\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.543924 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.543906 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5t8\" (UniqueName: \"kubernetes.io/projected/79d6dd09-2686-40ff-99c8-785c5a79116c-kube-api-access-sb5t8\") pod \"e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.652218 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.652103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:15.788695 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.788662 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4"] Jul 22 15:42:15.792374 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:42:15.792347 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d6dd09_2686_40ff_99c8_785c5a79116c.slice/crio-e58adb526ce9b8e4cba3a7b2e03ce5fcb96b1df43b598ccc942ee9efd29abaa6 WatchSource:0}: Error finding container e58adb526ce9b8e4cba3a7b2e03ce5fcb96b1df43b598ccc942ee9efd29abaa6: Status 404 returned error can't find the container with id e58adb526ce9b8e4cba3a7b2e03ce5fcb96b1df43b598ccc942ee9efd29abaa6 Jul 22 15:42:15.794650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.794604 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:42:15.794759 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.794692 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:42:15.794759 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.794734 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:42:15.990646 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.990563 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" event={"ID":"79d6dd09-2686-40ff-99c8-785c5a79116c","Type":"ContainerStarted","Data":"d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19"} Jul 22 15:42:15.990646 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:15.990613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" event={"ID":"79d6dd09-2686-40ff-99c8-785c5a79116c","Type":"ContainerStarted","Data":"e58adb526ce9b8e4cba3a7b2e03ce5fcb96b1df43b598ccc942ee9efd29abaa6"} Jul 22 15:42:16.013982 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:16.013925 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" podStartSLOduration=1.013906437 podStartE2EDuration="1.013906437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:42:16.012081168 +0000 UTC m=+2261.510980946" watchObservedRunningTime="2026-07-22 15:42:16.013906437 +0000 UTC m=+2261.512806215" Jul 22 15:42:16.653025 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:16.652991 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:16.657692 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:16.657669 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:16.995746 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:16.995667 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:16.996700 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:16.996677 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:19.710185 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.710162 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:42:19.773800 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.773719 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-workload-socket\") pod \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " Jul 22 15:42:19.773800 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.773760 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-envoy\") pod \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " Jul 22 15:42:19.773998 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.773815 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd7j9\" (UniqueName: \"kubernetes.io/projected/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-kube-api-access-qd7j9\") pod \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " Jul 22 15:42:19.773998 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.773860 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istiod-ca-cert\") pod \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " Jul 22 15:42:19.774209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.773985 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-podinfo\") pod \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " Jul 22 15:42:19.774209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.774017 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" (UID: "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:42:19.774209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.774031 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-data\") pod \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " Jul 22 15:42:19.774209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.774068 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-token\") pod \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " Jul 22 15:42:19.774209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.774106 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-workload-certs\") pod \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " Jul 22 15:42:19.774209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.774147 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-credential-socket\") pod \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\" (UID: \"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458\") " Jul 22 15:42:19.774511 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.774245 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" (UID: "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:42:19.774511 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.774492 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-workload-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:42:19.774607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.774499 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" (UID: "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:42:19.774607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.774516 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" (UID: "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:42:19.774607 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.774511 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istiod-ca-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:42:19.775059 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.775033 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" (UID: "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:42:19.776087 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.776062 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" (UID: "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:42:19.776336 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.776317 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" (UID: "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 15:42:19.776520 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.776501 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" (UID: "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:42:19.776520 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.776513 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-kube-api-access-qd7j9" (OuterVolumeSpecName: "kube-api-access-qd7j9") pod "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" (UID: "4d2d483f-1ab2-4e1c-bbba-ae5e1c804458"). InnerVolumeSpecName "kube-api-access-qd7j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:42:19.875298 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.875252 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-podinfo\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:42:19.875298 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.875291 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-data\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:42:19.875298 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.875299 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:42:19.875298 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.875307 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-workload-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:42:19.875298 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.875315 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-credential-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:42:19.875573 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.875324 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-istio-envoy\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:42:19.875573 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:19.875333 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd7j9\" (UniqueName: \"kubernetes.io/projected/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458-kube-api-access-qd7j9\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:42:20.009697 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:20.009663 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerID="daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc" exitCode=0 Jul 22 15:42:20.009886 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:20.009720 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" Jul 22 15:42:20.009886 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:20.009747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" event={"ID":"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458","Type":"ContainerDied","Data":"daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc"} Jul 22 15:42:20.009886 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:20.009800 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h" event={"ID":"4d2d483f-1ab2-4e1c-bbba-ae5e1c804458","Type":"ContainerDied","Data":"6e3248fef16e0c17c44fbd5702f600a1e80312ad262ab793ffc64fd990e8e5c0"} Jul 22 15:42:20.009886 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:20.009826 2580 scope.go:117] "RemoveContainer" containerID="daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc" Jul 22 15:42:20.018988 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:20.018970 2580 scope.go:117] "RemoveContainer" containerID="5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7" Jul 22 15:42:20.026651 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:20.026609 2580 scope.go:117] "RemoveContainer" containerID="daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc" Jul 22 15:42:20.026870 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:42:20.026851 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc\": container with ID starting with daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc not found: ID does not exist" containerID="daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc" Jul 22 15:42:20.026934 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:20.026879 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc"} err="failed to get container status \"daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc\": rpc error: code = NotFound desc = could not find container \"daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc\": container with ID starting with daa42bf4d8b34272c7fa1a11fc513e42d96a8acb37ca761e11e943888c0acbcc not found: ID does not exist" Jul 22 15:42:20.026934 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:20.026897 2580 scope.go:117] "RemoveContainer" containerID="5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7" Jul 22 15:42:20.027164 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:42:20.027145 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7\": container with ID starting with 5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7 not found: ID does not exist" containerID="5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7" Jul 22 15:42:20.027215 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:20.027170 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7"} err="failed to get container status \"5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7\": rpc error: code = NotFound desc = could not find container \"5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7\": container with ID starting with 5b2a682bacd4d3d71fde4f3badc6d7ed5604bb1f5903b2d762c5238c33ed95d7 not found: ID does not exist" Jul 22 15:42:20.035013 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:20.034992 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h"] Jul 22 15:42:20.039458 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:20.039436 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b47c9e-openshift-default-cd44bd665-ltk7h"] Jul 22 15:42:21.053769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:21.053732 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" path="/var/lib/kubelet/pods/4d2d483f-1ab2-4e1c-bbba-ae5e1c804458/volumes" Jul 22 15:42:23.177474 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.177441 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj"] Jul 22 15:42:23.177845 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.177791 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerName="istio-proxy" Jul 22 15:42:23.177845 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.177803 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerName="istio-proxy" Jul 22 15:42:23.177845 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.177812 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerName="istio-proxy" Jul 22 15:42:23.177845 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.177817 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerName="istio-proxy" Jul 22 15:42:23.177845 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.177837 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" Jul 22 15:42:23.177845 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.177842 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" Jul 22 15:42:23.178176 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.177940 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerName="istio-proxy" Jul 22 15:42:23.178176 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.177948 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d2d483f-1ab2-4e1c-bbba-ae5e1c804458" containerName="istio-proxy" Jul 22 15:42:23.178176 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.177956 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bbe6efd-789d-4b1a-98ff-2655eecf3575" containerName="istio-proxy" Jul 22 15:42:23.182555 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.182535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.185474 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.185456 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-552a8b-openshift-default-dockercfg-p9h6v\"" Jul 22 15:42:23.196513 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.196491 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj"] Jul 22 15:42:23.302908 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.302874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2777118f-3934-4227-af12-c6d8ccdedf22-istio-token\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.302908 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.302921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2777118f-3934-4227-af12-c6d8ccdedf22-istiod-ca-cert\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.303203 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.302947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-credential-socket\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.303203 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.302977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-workload-socket\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.303203 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.303003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-istio-envoy\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.303203 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.303050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-istio-data\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.303203 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.303084 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb65j\" (UniqueName: \"kubernetes.io/projected/2777118f-3934-4227-af12-c6d8ccdedf22-kube-api-access-kb65j\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.303203 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.303114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-workload-certs\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.303203 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.303168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2777118f-3934-4227-af12-c6d8ccdedf22-istio-podinfo\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404186 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-workload-certs\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404186 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404189 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2777118f-3934-4227-af12-c6d8ccdedf22-istio-podinfo\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404418 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2777118f-3934-4227-af12-c6d8ccdedf22-istio-token\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404418 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2777118f-3934-4227-af12-c6d8ccdedf22-istiod-ca-cert\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404418 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-credential-socket\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404418 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404378 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-workload-socket\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404418 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-istio-envoy\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404687 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-istio-data\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404687 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb65j\" (UniqueName: \"kubernetes.io/projected/2777118f-3934-4227-af12-c6d8ccdedf22-kube-api-access-kb65j\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404687 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404610 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-workload-certs\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404844 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404716 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-credential-socket\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404844 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404785 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-workload-socket\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.404984 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.404960 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-istio-data\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.405166 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.405147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2777118f-3934-4227-af12-c6d8ccdedf22-istiod-ca-cert\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.406601 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.406572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-istio-envoy\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.406703 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.406688 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2777118f-3934-4227-af12-c6d8ccdedf22-istio-podinfo\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.415566 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.415542 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2777118f-3934-4227-af12-c6d8ccdedf22-istio-token\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.415765 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.415742 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb65j\" (UniqueName: \"kubernetes.io/projected/2777118f-3934-4227-af12-c6d8ccdedf22-kube-api-access-kb65j\") pod \"e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.476416 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.476340 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:42:23.492872 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.492841 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:23.631261 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.631202 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj"] Jul 22 15:42:23.634882 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:42:23.634847 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2777118f_3934_4227_af12_c6d8ccdedf22.slice/crio-e3ddcd0dcb3f89036b143795f56f869d544c969f46c937e0285db4fa040e017c WatchSource:0}: Error finding container e3ddcd0dcb3f89036b143795f56f869d544c969f46c937e0285db4fa040e017c: Status 404 returned error can't find the container with id e3ddcd0dcb3f89036b143795f56f869d544c969f46c937e0285db4fa040e017c Jul 22 15:42:23.637567 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.637525 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:42:23.637686 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.637614 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:42:23.637686 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:23.637654 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:42:24.031589 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:24.031559 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" event={"ID":"2777118f-3934-4227-af12-c6d8ccdedf22","Type":"ContainerStarted","Data":"a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777"} Jul 22 15:42:24.031759 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:24.031605 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" event={"ID":"2777118f-3934-4227-af12-c6d8ccdedf22","Type":"ContainerStarted","Data":"e3ddcd0dcb3f89036b143795f56f869d544c969f46c937e0285db4fa040e017c"} Jul 22 15:42:24.053644 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:24.053597 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" podStartSLOduration=1.053584139 podStartE2EDuration="1.053584139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:42:24.051874999 +0000 UTC m=+2269.550774788" watchObservedRunningTime="2026-07-22 15:42:24.053584139 +0000 UTC m=+2269.552483915" Jul 22 15:42:24.493636 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:24.493606 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:24.498647 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:24.498620 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:25.040433 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:25.040398 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:25.041295 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:25.041273 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:31.996825 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:31.996785 2580 patch_prober.go:28] interesting pod/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 22 15:42:31.997260 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:31.996840 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 22 15:42:32.073547 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:32.073516 2580 generic.go:356] "Generic (PLEG): container finished" podID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerID="d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19" exitCode=0 Jul 22 15:42:32.073706 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:32.073584 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" event={"ID":"79d6dd09-2686-40ff-99c8-785c5a79116c","Type":"ContainerDied","Data":"d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19"} Jul 22 15:42:32.073988 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:32.073975 2580 scope.go:117] "RemoveContainer" containerID="d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19" Jul 22 15:42:32.074954 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:32.074926 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:42:32.075020 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:32.074994 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:42:32.075056 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:32.075032 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:42:33.082965 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:33.082917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" event={"ID":"79d6dd09-2686-40ff-99c8-785c5a79116c","Type":"ContainerStarted","Data":"7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604"} Jul 22 15:42:33.653354 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:33.652888 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:33.653354 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:33.653017 2580 patch_prober.go:28] interesting pod/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 22 15:42:33.653354 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:33.653071 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 22 15:42:34.653621 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:34.653581 2580 patch_prober.go:28] interesting pod/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 22 15:42:34.654156 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:34.653667 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 22 15:42:35.180754 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:35.180723 2580 scope.go:117] "RemoveContainer" containerID="128f5f7c8e7329332387aa3a2b2f9b90ed1e1cf4e2477241f407c3f2aa3afcc7" Jul 22 15:42:35.653182 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:35.653139 2580 patch_prober.go:28] interesting pod/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 22 15:42:35.653352 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:35.653207 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 22 15:42:36.652634 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:36.652588 2580 patch_prober.go:28] interesting pod/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 22 15:42:36.653054 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:36.652676 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 22 15:42:37.656375 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:37.656340 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:37.656764 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:37.656732 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:37.657359 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:37.657339 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:42:39.112648 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:39.112616 2580 generic.go:356] "Generic (PLEG): container finished" podID="2777118f-3934-4227-af12-c6d8ccdedf22" containerID="a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777" exitCode=0 Jul 22 15:42:39.113086 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:39.112697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" event={"ID":"2777118f-3934-4227-af12-c6d8ccdedf22","Type":"ContainerDied","Data":"a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777"} Jul 22 15:42:39.113158 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:39.113109 2580 scope.go:117] "RemoveContainer" containerID="a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777" Jul 22 15:42:39.114265 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:39.114232 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:42:39.114338 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:39.114320 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:42:39.114374 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:39.114363 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 22 15:42:39.275021 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:39.274987 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:42:40.040587 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:40.040551 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:40.119279 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:40.119251 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" event={"ID":"2777118f-3934-4227-af12-c6d8ccdedf22","Type":"ContainerStarted","Data":"f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121"} Jul 22 15:42:40.493020 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:40.492985 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:40.494093 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:40.494064 2580 patch_prober.go:28] interesting pod/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 15:42:40.494218 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:40.494140 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 15:42:41.493581 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:41.493546 2580 patch_prober.go:28] interesting pod/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 15:42:41.493940 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:41.493602 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 15:42:42.493909 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:42.493814 2580 patch_prober.go:28] interesting pod/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 15:42:42.493909 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:42.493870 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 15:42:43.493606 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:43.493564 2580 patch_prober.go:28] interesting pod/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 15:42:43.493816 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:43.493626 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 15:42:44.497018 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:44.496990 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:44.497465 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:44.497330 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:44.498049 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:44.498033 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:42:48.581227 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:48.581198 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:42:59.792779 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.792743 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd"] Jul 22 15:42:59.796069 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.796051 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:42:59.801549 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.801523 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-552a8b\"/\"kube-root-ca.crt\"" Jul 22 15:42:59.802953 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.802925 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-552a8b\"/\"auth-test-model-552a8b-kserve-self-signed-certs\"" Jul 22 15:42:59.803094 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.802970 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-552a8b\"/\"default-dockercfg-lmhdp\"" Jul 22 15:42:59.817049 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.817022 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd"] Jul 22 15:42:59.834512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.834488 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-552a8b\"/\"openshift-service-ca.crt\"" Jul 22 15:42:59.937285 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.937253 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725a8477-a371-4ad1-889f-d8b72e298218-tls-certs\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:42:59.937457 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.937294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-tmp-dir\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:42:59.937457 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.937325 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-dshm\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:42:59.937457 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.937382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-home\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:42:59.937457 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.937427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-model-cache\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:42:59.937457 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:42:59.937445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hglwn\" (UniqueName: \"kubernetes.io/projected/725a8477-a371-4ad1-889f-d8b72e298218-kube-api-access-hglwn\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.038825 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.038786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725a8477-a371-4ad1-889f-d8b72e298218-tls-certs\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.038825 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.038830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-tmp-dir\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.039074 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.038869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-dshm\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.039074 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.038900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-home\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.039074 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.038944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-model-cache\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.039074 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.038966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglwn\" (UniqueName: \"kubernetes.io/projected/725a8477-a371-4ad1-889f-d8b72e298218-kube-api-access-hglwn\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.039407 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.039372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-tmp-dir\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.039527 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.039407 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-home\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.039527 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.039466 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-model-cache\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.041251 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.041228 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-dshm\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.041454 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.041437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725a8477-a371-4ad1-889f-d8b72e298218-tls-certs\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.056315 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.056250 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hglwn\" (UniqueName: \"kubernetes.io/projected/725a8477-a371-4ad1-889f-d8b72e298218-kube-api-access-hglwn\") pod \"auth-test-model-552a8b-kserve-6c845749f5-qbxbd\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.105210 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.105185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:00.241051 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:00.241020 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd"] Jul 22 15:43:00.243287 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:43:00.243259 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod725a8477_a371_4ad1_889f_d8b72e298218.slice/crio-695aed9a75e55c7debfbf17edce49b8fb3d3522faa070442d762ca59057b960d WatchSource:0}: Error finding container 695aed9a75e55c7debfbf17edce49b8fb3d3522faa070442d762ca59057b960d: Status 404 returned error can't find the container with id 695aed9a75e55c7debfbf17edce49b8fb3d3522faa070442d762ca59057b960d Jul 22 15:43:01.229985 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:01.229951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" event={"ID":"725a8477-a371-4ad1-889f-d8b72e298218","Type":"ContainerStarted","Data":"7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312"} Jul 22 15:43:01.230384 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:01.229998 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" event={"ID":"725a8477-a371-4ad1-889f-d8b72e298218","Type":"ContainerStarted","Data":"695aed9a75e55c7debfbf17edce49b8fb3d3522faa070442d762ca59057b960d"} Jul 22 15:43:01.255025 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:01.254976 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" podStartSLOduration=2.254961797 podStartE2EDuration="2.254961797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:43:01.252648046 +0000 UTC m=+2306.751547822" watchObservedRunningTime="2026-07-22 15:43:01.254961797 +0000 UTC m=+2306.753861572" Jul 22 15:43:03.674036 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:03.674003 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:43:10.106002 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:10.105972 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:10.106387 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:10.106238 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:10.122240 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:10.122219 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:10.290091 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:10.290061 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:43:17.379983 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:17.379951 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:43:35.569299 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:35.569223 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:43:50.079302 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:43:50.079273 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:44:07.374721 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:44:07.374680 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:44:25.278022 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:44:25.277991 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:44:43.975994 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:44:43.975965 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:44:56.022879 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:44:56.022846 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd"] Jul 22 15:44:58.781746 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:44:58.781668 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:45:11.042963 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.042901 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" podUID="725a8477-a371-4ad1-889f-d8b72e298218" containerName="main" containerID="cri-o://7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312" gracePeriod=45 Jul 22 15:45:11.295769 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.295713 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:45:11.341871 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.341844 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-tmp-dir\") pod \"725a8477-a371-4ad1-889f-d8b72e298218\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " Jul 22 15:45:11.342041 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.341890 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hglwn\" (UniqueName: \"kubernetes.io/projected/725a8477-a371-4ad1-889f-d8b72e298218-kube-api-access-hglwn\") pod \"725a8477-a371-4ad1-889f-d8b72e298218\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " Jul 22 15:45:11.342041 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.341924 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725a8477-a371-4ad1-889f-d8b72e298218-tls-certs\") pod \"725a8477-a371-4ad1-889f-d8b72e298218\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " Jul 22 15:45:11.342041 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.341939 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-dshm\") pod \"725a8477-a371-4ad1-889f-d8b72e298218\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " Jul 22 15:45:11.342041 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.341988 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-model-cache\") pod \"725a8477-a371-4ad1-889f-d8b72e298218\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " Jul 22 15:45:11.342041 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.342005 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-home\") pod \"725a8477-a371-4ad1-889f-d8b72e298218\" (UID: \"725a8477-a371-4ad1-889f-d8b72e298218\") " Jul 22 15:45:11.342338 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.342171 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "725a8477-a371-4ad1-889f-d8b72e298218" (UID: "725a8477-a371-4ad1-889f-d8b72e298218"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:45:11.342338 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.342260 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-model-cache" (OuterVolumeSpecName: "model-cache") pod "725a8477-a371-4ad1-889f-d8b72e298218" (UID: "725a8477-a371-4ad1-889f-d8b72e298218"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:45:11.342338 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.342317 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-home" (OuterVolumeSpecName: "home") pod "725a8477-a371-4ad1-889f-d8b72e298218" (UID: "725a8477-a371-4ad1-889f-d8b72e298218"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:45:11.344230 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.344208 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-dshm" (OuterVolumeSpecName: "dshm") pod "725a8477-a371-4ad1-889f-d8b72e298218" (UID: "725a8477-a371-4ad1-889f-d8b72e298218"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:45:11.344230 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.344215 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725a8477-a371-4ad1-889f-d8b72e298218-kube-api-access-hglwn" (OuterVolumeSpecName: "kube-api-access-hglwn") pod "725a8477-a371-4ad1-889f-d8b72e298218" (UID: "725a8477-a371-4ad1-889f-d8b72e298218"). InnerVolumeSpecName "kube-api-access-hglwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:45:11.344351 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.344286 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/725a8477-a371-4ad1-889f-d8b72e298218-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "725a8477-a371-4ad1-889f-d8b72e298218" (UID: "725a8477-a371-4ad1-889f-d8b72e298218"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 15:45:11.443099 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.443064 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-model-cache\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:45:11.443099 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.443092 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-home\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:45:11.443307 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.443144 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-tmp-dir\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:45:11.443307 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.443154 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hglwn\" (UniqueName: \"kubernetes.io/projected/725a8477-a371-4ad1-889f-d8b72e298218-kube-api-access-hglwn\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:45:11.443307 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.443163 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725a8477-a371-4ad1-889f-d8b72e298218-tls-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:45:11.443307 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.443172 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725a8477-a371-4ad1-889f-d8b72e298218-dshm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:45:11.820997 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.820965 2580 generic.go:356] "Generic (PLEG): container finished" podID="725a8477-a371-4ad1-889f-d8b72e298218" containerID="7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312" exitCode=0 Jul 22 15:45:11.821226 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.821009 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" Jul 22 15:45:11.821226 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.821046 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" event={"ID":"725a8477-a371-4ad1-889f-d8b72e298218","Type":"ContainerDied","Data":"7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312"} Jul 22 15:45:11.821226 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.821078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd" event={"ID":"725a8477-a371-4ad1-889f-d8b72e298218","Type":"ContainerDied","Data":"695aed9a75e55c7debfbf17edce49b8fb3d3522faa070442d762ca59057b960d"} Jul 22 15:45:11.821226 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.821106 2580 scope.go:117] "RemoveContainer" containerID="7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312" Jul 22 15:45:11.830631 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.830608 2580 scope.go:117] "RemoveContainer" containerID="7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312" Jul 22 15:45:11.830902 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:45:11.830879 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312\": container with ID starting with 7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312 not found: ID does not exist" containerID="7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312" Jul 22 15:45:11.830975 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.830910 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312"} err="failed to get container status \"7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312\": rpc error: code = NotFound desc = could not find container \"7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312\": container with ID starting with 7812cbf9d3bf6b69abcadfd2205170a9beae905e171044d51f671b83a4fbb312 not found: ID does not exist" Jul 22 15:45:11.842846 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.842826 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd"] Jul 22 15:45:11.846777 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:11.846754 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/auth-test-model-552a8b-kserve-6c845749f5-qbxbd"] Jul 22 15:45:12.272283 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:12.272200 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:45:13.053770 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:13.053731 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="725a8477-a371-4ad1-889f-d8b72e298218" path="/var/lib/kubelet/pods/725a8477-a371-4ad1-889f-d8b72e298218/volumes" Jul 22 15:45:45.987077 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:45.987046 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9"] Jul 22 15:45:45.987529 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:45.987407 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725a8477-a371-4ad1-889f-d8b72e298218" containerName="main" Jul 22 15:45:45.987529 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:45.987418 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="725a8477-a371-4ad1-889f-d8b72e298218" containerName="main" Jul 22 15:45:45.987529 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:45.987469 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="725a8477-a371-4ad1-889f-d8b72e298218" containerName="main" Jul 22 15:45:45.990414 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:45.990399 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:45.993499 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:45.993479 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-552a8b\"/\"sub-test-model-552a8b-kserve-self-signed-certs\"" Jul 22 15:45:45.994842 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:45.994820 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-552a8b\"/\"kube-root-ca.crt\"" Jul 22 15:45:45.994970 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:45.994874 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-552a8b\"/\"default-dockercfg-lmhdp\"" Jul 22 15:45:45.994970 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:45.994876 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-552a8b\"/\"openshift-service-ca.crt\"" Jul 22 15:45:45.999821 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:45.999800 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9"] Jul 22 15:45:46.124004 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.123966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-model-cache\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.124209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.124035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-home\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.124209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.124061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-dshm\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.124209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.124084 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-tmp-dir\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.124209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.124198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-tls-certs\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.124378 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.124269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jksq\" (UniqueName: \"kubernetes.io/projected/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-kube-api-access-4jksq\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.225293 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.225257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-home\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.225293 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.225293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-dshm\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.225532 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.225317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-tmp-dir\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.225532 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.225375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-tls-certs\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.225532 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.225440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jksq\" (UniqueName: \"kubernetes.io/projected/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-kube-api-access-4jksq\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.225532 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.225525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-model-cache\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.225873 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.225848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-home\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.225979 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.225885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-model-cache\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.225979 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.225950 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-tmp-dir\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.227591 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.227572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-dshm\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.227872 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.227855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-tls-certs\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.233839 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.233806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jksq\" (UniqueName: \"kubernetes.io/projected/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-kube-api-access-4jksq\") pod \"sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.300428 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.300347 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:46.460993 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.460963 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9"] Jul 22 15:45:46.465978 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:45:46.465938 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fb8aee5_ac81_4625_a24a_3b5dd18d4d7f.slice/crio-c3ad2897c483b191db30e8d6f379e035a40e6dc567695353d02233cedcb569fb WatchSource:0}: Error finding container c3ad2897c483b191db30e8d6f379e035a40e6dc567695353d02233cedcb569fb: Status 404 returned error can't find the container with id c3ad2897c483b191db30e8d6f379e035a40e6dc567695353d02233cedcb569fb Jul 22 15:45:46.985995 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.985961 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" event={"ID":"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f","Type":"ContainerStarted","Data":"dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a"} Jul 22 15:45:46.985995 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:46.986009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" event={"ID":"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f","Type":"ContainerStarted","Data":"c3ad2897c483b191db30e8d6f379e035a40e6dc567695353d02233cedcb569fb"} Jul 22 15:45:47.005527 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:47.005479 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" podStartSLOduration=2.005464778 podStartE2EDuration="2.005464778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:45:47.004963909 +0000 UTC m=+2472.503863686" watchObservedRunningTime="2026-07-22 15:45:47.005464778 +0000 UTC m=+2472.504364553" Jul 22 15:45:49.783532 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:49.783495 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:45:56.301152 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:56.301097 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:56.301152 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:56.301167 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:56.313835 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:56.313807 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:45:57.047088 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:45:57.047060 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:46:03.978295 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:03.978263 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:46:24.385397 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:24.385364 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:46:32.375453 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:32.375370 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9"] Jul 22 15:46:35.229744 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:35.229711 2580 scope.go:117] "RemoveContainer" containerID="932c7d73cea53e24d2eafd7aa161f190d3ee52d88a54be0f91ba5f2c8e0f3cb5" Jul 22 15:46:41.781233 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:41.781200 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:46:47.394992 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.394953 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" podUID="8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f" containerName="main" containerID="cri-o://dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a" gracePeriod=45 Jul 22 15:46:47.654073 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.654017 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:46:47.755752 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.755719 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-tmp-dir\") pod \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " Jul 22 15:46:47.755927 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.755790 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-home\") pod \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " Jul 22 15:46:47.755927 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.755819 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-model-cache\") pod \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " Jul 22 15:46:47.755927 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.755846 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jksq\" (UniqueName: \"kubernetes.io/projected/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-kube-api-access-4jksq\") pod \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " Jul 22 15:46:47.755927 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.755917 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-dshm\") pod \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " Jul 22 15:46:47.756173 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.755955 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-tls-certs\") pod \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\" (UID: \"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f\") " Jul 22 15:46:47.756173 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.756020 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f" (UID: "8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:46:47.756173 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.756059 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-home" (OuterVolumeSpecName: "home") pod "8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f" (UID: "8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:46:47.756173 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.756097 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f" (UID: "8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:46:47.756395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.756242 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-tmp-dir\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:46:47.756395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.756254 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-home\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:46:47.756395 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.756263 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-model-cache\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:46:47.757983 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.757960 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f" (UID: "8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 15:46:47.758100 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.757978 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-kube-api-access-4jksq" (OuterVolumeSpecName: "kube-api-access-4jksq") pod "8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f" (UID: "8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f"). InnerVolumeSpecName "kube-api-access-4jksq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:46:47.758100 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.757999 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-dshm" (OuterVolumeSpecName: "dshm") pod "8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f" (UID: "8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:46:47.856880 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.856848 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-tls-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:46:47.856880 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.856878 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jksq\" (UniqueName: \"kubernetes.io/projected/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-kube-api-access-4jksq\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:46:47.856880 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:47.856887 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f-dshm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:46:48.275566 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:48.275533 2580 generic.go:356] "Generic (PLEG): container finished" podID="8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f" containerID="dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a" exitCode=0 Jul 22 15:46:48.275772 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:48.275576 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" Jul 22 15:46:48.275772 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:48.275627 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" event={"ID":"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f","Type":"ContainerDied","Data":"dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a"} Jul 22 15:46:48.275772 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:48.275679 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9" event={"ID":"8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f","Type":"ContainerDied","Data":"c3ad2897c483b191db30e8d6f379e035a40e6dc567695353d02233cedcb569fb"} Jul 22 15:46:48.275772 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:48.275703 2580 scope.go:117] "RemoveContainer" containerID="dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a" Jul 22 15:46:48.284523 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:48.284510 2580 scope.go:117] "RemoveContainer" containerID="dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a" Jul 22 15:46:48.284751 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:46:48.284733 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a\": container with ID starting with dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a not found: ID does not exist" containerID="dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a" Jul 22 15:46:48.284808 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:48.284759 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a"} err="failed to get container status \"dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a\": rpc error: code = NotFound desc = could not find container \"dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a\": container with ID starting with dc571465bdbc4faa0f2e1e58550545e8926be2b14ddc77b5b2f0db4089c4f31a not found: ID does not exist" Jul 22 15:46:48.297770 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:48.297745 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9"] Jul 22 15:46:48.302023 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:48.302003 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/sub-test-model-552a8b-kserve-9dd7bdbf-dg8q9"] Jul 22 15:46:49.054646 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:49.054613 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f" path="/var/lib/kubelet/pods/8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f/volumes" Jul 22 15:46:51.571740 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:46:51.571704 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:47:07.785823 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.785789 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df"] Jul 22 15:47:07.786320 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.786201 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f" containerName="main" Jul 22 15:47:07.786320 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.786218 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f" containerName="main" Jul 22 15:47:07.786320 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.786278 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fb8aee5-ac81-4625-a24a-3b5dd18d4d7f" containerName="main" Jul 22 15:47:07.790742 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.790722 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.794195 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.794174 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-552a8b\"/\"kube-root-ca.crt\"" Jul 22 15:47:07.794318 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.794210 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-552a8b\"/\"openshift-service-ca.crt\"" Jul 22 15:47:07.795472 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.795451 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-552a8b\"/\"rate-test-model-552a8b-kserve-self-signed-certs\"" Jul 22 15:47:07.795472 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.795471 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-552a8b\"/\"default-dockercfg-lmhdp\"" Jul 22 15:47:07.799199 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.799176 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df"] Jul 22 15:47:07.805847 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.805828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5qjx\" (UniqueName: \"kubernetes.io/projected/47fc0362-884d-4e4b-a335-8b392f2d2697-kube-api-access-f5qjx\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.805945 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.805860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fc0362-884d-4e4b-a335-8b392f2d2697-tls-certs\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.805945 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.805888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-home\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.806020 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.805953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-model-cache\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.806020 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.806012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-tmp-dir\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.806091 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.806071 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-dshm\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.907048 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.907020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-home\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.907048 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.907050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-model-cache\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.907293 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.907075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-tmp-dir\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.907293 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.907106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-dshm\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.907293 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.907168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5qjx\" (UniqueName: \"kubernetes.io/projected/47fc0362-884d-4e4b-a335-8b392f2d2697-kube-api-access-f5qjx\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.907293 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.907196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fc0362-884d-4e4b-a335-8b392f2d2697-tls-certs\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.907519 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.907433 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-tmp-dir\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.907519 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.907504 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-home\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.907602 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.907568 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-model-cache\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.909393 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.909373 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-dshm\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.909563 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.909545 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fc0362-884d-4e4b-a335-8b392f2d2697-tls-certs\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:07.916849 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:07.916821 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5qjx\" (UniqueName: \"kubernetes.io/projected/47fc0362-884d-4e4b-a335-8b392f2d2697-kube-api-access-f5qjx\") pod \"rate-test-model-552a8b-kserve-6b98f848b7-fw4df\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:08.100699 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:08.100666 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:08.237956 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:08.237924 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df"] Jul 22 15:47:08.240836 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:47:08.240797 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47fc0362_884d_4e4b_a335_8b392f2d2697.slice/crio-29e5f8d97f02ca698d1f6a9b799c44d12c0ce2afdf42ac515169d75b54773477 WatchSource:0}: Error finding container 29e5f8d97f02ca698d1f6a9b799c44d12c0ce2afdf42ac515169d75b54773477: Status 404 returned error can't find the container with id 29e5f8d97f02ca698d1f6a9b799c44d12c0ce2afdf42ac515169d75b54773477 Jul 22 15:47:08.368112 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:08.368036 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" event={"ID":"47fc0362-884d-4e4b-a335-8b392f2d2697","Type":"ContainerStarted","Data":"9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066"} Jul 22 15:47:08.368112 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:08.368080 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" event={"ID":"47fc0362-884d-4e4b-a335-8b392f2d2697","Type":"ContainerStarted","Data":"29e5f8d97f02ca698d1f6a9b799c44d12c0ce2afdf42ac515169d75b54773477"} Jul 22 15:47:08.392813 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:08.392755 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" podStartSLOduration=1.39273881 podStartE2EDuration="1.39273881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 15:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:47:08.391494041 +0000 UTC m=+2553.890393817" watchObservedRunningTime="2026-07-22 15:47:08.39273881 +0000 UTC m=+2553.891638586" Jul 22 15:47:11.284390 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:11.284352 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:47:18.101197 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:18.101159 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:18.101573 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:18.101211 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:18.113696 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:18.113675 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:18.435327 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:18.435242 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:47:25.593493 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:25.593461 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:47:45.367209 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:45.367180 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:47:52.969624 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:52.969584 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df"] Jul 22 15:47:59.470328 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:47:59.470249 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:48:07.988766 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:07.988706 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" podUID="47fc0362-884d-4e4b-a335-8b392f2d2697" containerName="main" containerID="cri-o://9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066" gracePeriod=45 Jul 22 15:48:08.235576 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.235544 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:48:08.331666 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.331642 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-home\") pod \"47fc0362-884d-4e4b-a335-8b392f2d2697\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " Jul 22 15:48:08.331836 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.331694 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-model-cache\") pod \"47fc0362-884d-4e4b-a335-8b392f2d2697\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " Jul 22 15:48:08.331836 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.331709 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-dshm\") pod \"47fc0362-884d-4e4b-a335-8b392f2d2697\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " Jul 22 15:48:08.331836 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.331737 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5qjx\" (UniqueName: \"kubernetes.io/projected/47fc0362-884d-4e4b-a335-8b392f2d2697-kube-api-access-f5qjx\") pod \"47fc0362-884d-4e4b-a335-8b392f2d2697\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " Jul 22 15:48:08.331836 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.331767 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-tmp-dir\") pod \"47fc0362-884d-4e4b-a335-8b392f2d2697\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " Jul 22 15:48:08.331836 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.331811 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fc0362-884d-4e4b-a335-8b392f2d2697-tls-certs\") pod \"47fc0362-884d-4e4b-a335-8b392f2d2697\" (UID: \"47fc0362-884d-4e4b-a335-8b392f2d2697\") " Jul 22 15:48:08.332107 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.331923 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-home" (OuterVolumeSpecName: "home") pod "47fc0362-884d-4e4b-a335-8b392f2d2697" (UID: "47fc0362-884d-4e4b-a335-8b392f2d2697"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:48:08.332107 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.331936 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-model-cache" (OuterVolumeSpecName: "model-cache") pod "47fc0362-884d-4e4b-a335-8b392f2d2697" (UID: "47fc0362-884d-4e4b-a335-8b392f2d2697"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:48:08.332107 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.332092 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-home\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:08.332262 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.332111 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-model-cache\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:08.332262 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.332103 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47fc0362-884d-4e4b-a335-8b392f2d2697" (UID: "47fc0362-884d-4e4b-a335-8b392f2d2697"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:48:08.333861 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.333840 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47fc0362-884d-4e4b-a335-8b392f2d2697-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47fc0362-884d-4e4b-a335-8b392f2d2697" (UID: "47fc0362-884d-4e4b-a335-8b392f2d2697"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 15:48:08.334241 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.334216 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fc0362-884d-4e4b-a335-8b392f2d2697-kube-api-access-f5qjx" (OuterVolumeSpecName: "kube-api-access-f5qjx") pod "47fc0362-884d-4e4b-a335-8b392f2d2697" (UID: "47fc0362-884d-4e4b-a335-8b392f2d2697"). InnerVolumeSpecName "kube-api-access-f5qjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:48:08.334324 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.334308 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-dshm" (OuterVolumeSpecName: "dshm") pod "47fc0362-884d-4e4b-a335-8b392f2d2697" (UID: "47fc0362-884d-4e4b-a335-8b392f2d2697"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:48:08.433092 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.433063 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5qjx\" (UniqueName: \"kubernetes.io/projected/47fc0362-884d-4e4b-a335-8b392f2d2697-kube-api-access-f5qjx\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:08.433244 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.433106 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-tmp-dir\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:08.433244 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.433138 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fc0362-884d-4e4b-a335-8b392f2d2697-tls-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:08.433244 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.433148 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47fc0362-884d-4e4b-a335-8b392f2d2697-dshm\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:08.657339 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.657253 2580 generic.go:356] "Generic (PLEG): container finished" podID="47fc0362-884d-4e4b-a335-8b392f2d2697" containerID="9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066" exitCode=0 Jul 22 15:48:08.657339 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.657298 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" Jul 22 15:48:08.657512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.657347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" event={"ID":"47fc0362-884d-4e4b-a335-8b392f2d2697","Type":"ContainerDied","Data":"9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066"} Jul 22 15:48:08.657512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.657393 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df" event={"ID":"47fc0362-884d-4e4b-a335-8b392f2d2697","Type":"ContainerDied","Data":"29e5f8d97f02ca698d1f6a9b799c44d12c0ce2afdf42ac515169d75b54773477"} Jul 22 15:48:08.657512 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.657412 2580 scope.go:117] "RemoveContainer" containerID="9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066" Jul 22 15:48:08.666291 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.666274 2580 scope.go:117] "RemoveContainer" containerID="9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066" Jul 22 15:48:08.666536 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:48:08.666519 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066\": container with ID starting with 9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066 not found: ID does not exist" containerID="9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066" Jul 22 15:48:08.666583 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.666545 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066"} err="failed to get container status \"9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066\": rpc error: code = NotFound desc = could not find container \"9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066\": container with ID starting with 9952ddf343587b9bd355edddc2993b997b732a235dbd68ffd04addc12e9af066 not found: ID does not exist" Jul 22 15:48:08.680417 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.680395 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df"] Jul 22 15:48:08.689948 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:08.689928 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-552a8b/rate-test-model-552a8b-kserve-6b98f848b7-fw4df"] Jul 22 15:48:09.053772 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:09.053699 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47fc0362-884d-4e4b-a335-8b392f2d2697" path="/var/lib/kubelet/pods/47fc0362-884d-4e4b-a335-8b392f2d2697/volumes" Jul 22 15:48:26.712394 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:26.712358 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:48:34.927347 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:34.927309 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4"] Jul 22 15:48:34.927849 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:34.927556 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" containerID="cri-o://7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604" gracePeriod=30 Jul 22 15:48:37.659193 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:37.659155 2580 patch_prober.go:28] interesting pod/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 15:48:37.659574 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:37.659226 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 15:48:40.181081 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.181055 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:48:40.296111 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296023 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-workload-socket\") pod \"79d6dd09-2686-40ff-99c8-785c5a79116c\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " Jul 22 15:48:40.296111 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296076 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-envoy\") pod \"79d6dd09-2686-40ff-99c8-785c5a79116c\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " Jul 22 15:48:40.296111 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296107 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-data\") pod \"79d6dd09-2686-40ff-99c8-785c5a79116c\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " Jul 22 15:48:40.296423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296195 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-credential-socket\") pod \"79d6dd09-2686-40ff-99c8-785c5a79116c\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " Jul 22 15:48:40.296423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296224 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-podinfo\") pod \"79d6dd09-2686-40ff-99c8-785c5a79116c\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " Jul 22 15:48:40.296423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296262 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb5t8\" (UniqueName: \"kubernetes.io/projected/79d6dd09-2686-40ff-99c8-785c5a79116c-kube-api-access-sb5t8\") pod \"79d6dd09-2686-40ff-99c8-785c5a79116c\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " Jul 22 15:48:40.296423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296290 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-workload-certs\") pod \"79d6dd09-2686-40ff-99c8-785c5a79116c\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " Jul 22 15:48:40.296423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296335 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d6dd09-2686-40ff-99c8-785c5a79116c-istiod-ca-cert\") pod \"79d6dd09-2686-40ff-99c8-785c5a79116c\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " Jul 22 15:48:40.296423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296339 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79d6dd09-2686-40ff-99c8-785c5a79116c" (UID: "79d6dd09-2686-40ff-99c8-785c5a79116c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:48:40.296423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296365 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-token\") pod \"79d6dd09-2686-40ff-99c8-785c5a79116c\" (UID: \"79d6dd09-2686-40ff-99c8-785c5a79116c\") " Jul 22 15:48:40.296775 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296510 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79d6dd09-2686-40ff-99c8-785c5a79116c" (UID: "79d6dd09-2686-40ff-99c8-785c5a79116c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:48:40.296775 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296550 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79d6dd09-2686-40ff-99c8-785c5a79116c" (UID: "79d6dd09-2686-40ff-99c8-785c5a79116c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:48:40.296775 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296618 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-credential-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:40.296775 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296634 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-workload-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:40.296775 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296648 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-workload-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:40.297099 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.296787 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d6dd09-2686-40ff-99c8-785c5a79116c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79d6dd09-2686-40ff-99c8-785c5a79116c" (UID: "79d6dd09-2686-40ff-99c8-785c5a79116c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:48:40.297191 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.297131 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-data" (OuterVolumeSpecName: "istio-data") pod "79d6dd09-2686-40ff-99c8-785c5a79116c" (UID: "79d6dd09-2686-40ff-99c8-785c5a79116c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:48:40.298467 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.298443 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79d6dd09-2686-40ff-99c8-785c5a79116c" (UID: "79d6dd09-2686-40ff-99c8-785c5a79116c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:48:40.298582 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.298468 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d6dd09-2686-40ff-99c8-785c5a79116c-kube-api-access-sb5t8" (OuterVolumeSpecName: "kube-api-access-sb5t8") pod "79d6dd09-2686-40ff-99c8-785c5a79116c" (UID: "79d6dd09-2686-40ff-99c8-785c5a79116c"). InnerVolumeSpecName "kube-api-access-sb5t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:48:40.298777 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.298754 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79d6dd09-2686-40ff-99c8-785c5a79116c" (UID: "79d6dd09-2686-40ff-99c8-785c5a79116c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 15:48:40.298777 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.298750 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-token" (OuterVolumeSpecName: "istio-token") pod "79d6dd09-2686-40ff-99c8-785c5a79116c" (UID: "79d6dd09-2686-40ff-99c8-785c5a79116c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:48:40.397429 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.397380 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb5t8\" (UniqueName: \"kubernetes.io/projected/79d6dd09-2686-40ff-99c8-785c5a79116c-kube-api-access-sb5t8\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:40.397429 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.397427 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d6dd09-2686-40ff-99c8-785c5a79116c-istiod-ca-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:40.397429 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.397437 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:40.397429 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.397445 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-envoy\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:40.397704 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.397453 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-data\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:40.397704 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.397461 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d6dd09-2686-40ff-99c8-785c5a79116c-istio-podinfo\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:48:40.803967 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.803937 2580 generic.go:356] "Generic (PLEG): container finished" podID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerID="7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604" exitCode=0 Jul 22 15:48:40.804190 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.803988 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" Jul 22 15:48:40.804190 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.804012 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" event={"ID":"79d6dd09-2686-40ff-99c8-785c5a79116c","Type":"ContainerDied","Data":"7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604"} Jul 22 15:48:40.804190 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.804072 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4" event={"ID":"79d6dd09-2686-40ff-99c8-785c5a79116c","Type":"ContainerDied","Data":"e58adb526ce9b8e4cba3a7b2e03ce5fcb96b1df43b598ccc942ee9efd29abaa6"} Jul 22 15:48:40.804190 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.804093 2580 scope.go:117] "RemoveContainer" containerID="7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604" Jul 22 15:48:40.813331 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.813313 2580 scope.go:117] "RemoveContainer" containerID="d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19" Jul 22 15:48:40.821040 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.821020 2580 scope.go:117] "RemoveContainer" containerID="7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604" Jul 22 15:48:40.821297 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:48:40.821279 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604\": container with ID starting with 7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604 not found: ID does not exist" containerID="7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604" Jul 22 15:48:40.821356 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.821304 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604"} err="failed to get container status \"7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604\": rpc error: code = NotFound desc = could not find container \"7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604\": container with ID starting with 7892a5c982e5495f2bcdd548e599a2a3bdd263f27e53824b81f88a94ce5d6604 not found: ID does not exist" Jul 22 15:48:40.821356 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.821322 2580 scope.go:117] "RemoveContainer" containerID="d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19" Jul 22 15:48:40.821554 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:48:40.821535 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19\": container with ID starting with d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19 not found: ID does not exist" containerID="d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19" Jul 22 15:48:40.821608 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.821565 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19"} err="failed to get container status \"d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19\": rpc error: code = NotFound desc = could not find container \"d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19\": container with ID starting with d4f22078f9c0c0b25fdf1f72a7c13a3df6c2dbefba3948e3110c7de6fb42ab19 not found: ID does not exist" Jul 22 15:48:40.830590 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.830566 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4"] Jul 22 15:48:40.835717 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:40.835695 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-6f1c00-openshift-default-54f5576cff-4f7q4"] Jul 22 15:48:41.053856 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:41.053821 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" path="/var/lib/kubelet/pods/79d6dd09-2686-40ff-99c8-785c5a79116c/volumes" Jul 22 15:48:48.574112 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:48.574071 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d697m"] Jul 22 15:48:58.092239 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:58.092192 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj"] Jul 22 15:48:58.092645 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:58.092578 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" containerName="istio-proxy" containerID="cri-o://f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121" gracePeriod=30 Jul 22 15:48:59.499645 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:59.499606 2580 patch_prober.go:28] interesting pod/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 15:48:59.500015 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:48:59.499686 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 15:49:03.351000 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.350975 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:49:03.380225 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.380146 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb65j\" (UniqueName: \"kubernetes.io/projected/2777118f-3934-4227-af12-c6d8ccdedf22-kube-api-access-kb65j\") pod \"2777118f-3934-4227-af12-c6d8ccdedf22\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " Jul 22 15:49:03.380225 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.380187 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2777118f-3934-4227-af12-c6d8ccdedf22-istio-podinfo\") pod \"2777118f-3934-4227-af12-c6d8ccdedf22\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " Jul 22 15:49:03.380225 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.380209 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2777118f-3934-4227-af12-c6d8ccdedf22-istio-token\") pod \"2777118f-3934-4227-af12-c6d8ccdedf22\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " Jul 22 15:49:03.380448 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.380240 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2777118f-3934-4227-af12-c6d8ccdedf22-istiod-ca-cert\") pod \"2777118f-3934-4227-af12-c6d8ccdedf22\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " Jul 22 15:49:03.380448 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.380329 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-istio-data\") pod \"2777118f-3934-4227-af12-c6d8ccdedf22\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " Jul 22 15:49:03.380448 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.380381 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-workload-socket\") pod \"2777118f-3934-4227-af12-c6d8ccdedf22\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " Jul 22 15:49:03.380448 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.380433 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-workload-certs\") pod \"2777118f-3934-4227-af12-c6d8ccdedf22\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " Jul 22 15:49:03.380723 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.380679 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2777118f-3934-4227-af12-c6d8ccdedf22-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2777118f-3934-4227-af12-c6d8ccdedf22" (UID: "2777118f-3934-4227-af12-c6d8ccdedf22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 15:49:03.380877 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.380807 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2777118f-3934-4227-af12-c6d8ccdedf22" (UID: "2777118f-3934-4227-af12-c6d8ccdedf22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:49:03.380877 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.380863 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2777118f-3934-4227-af12-c6d8ccdedf22" (UID: "2777118f-3934-4227-af12-c6d8ccdedf22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:49:03.381004 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.380923 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-credential-socket\") pod \"2777118f-3934-4227-af12-c6d8ccdedf22\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " Jul 22 15:49:03.381004 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.380950 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-istio-envoy\") pod \"2777118f-3934-4227-af12-c6d8ccdedf22\" (UID: \"2777118f-3934-4227-af12-c6d8ccdedf22\") " Jul 22 15:49:03.381255 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.381222 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2777118f-3934-4227-af12-c6d8ccdedf22" (UID: "2777118f-3934-4227-af12-c6d8ccdedf22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:49:03.381369 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.381270 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2777118f-3934-4227-af12-c6d8ccdedf22-istiod-ca-cert\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:49:03.381369 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.381292 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-workload-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:49:03.381369 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.381306 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-workload-certs\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:49:03.381541 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.381435 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-istio-data" (OuterVolumeSpecName: "istio-data") pod "2777118f-3934-4227-af12-c6d8ccdedf22" (UID: "2777118f-3934-4227-af12-c6d8ccdedf22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:49:03.382546 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.382521 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2777118f-3934-4227-af12-c6d8ccdedf22-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2777118f-3934-4227-af12-c6d8ccdedf22" (UID: "2777118f-3934-4227-af12-c6d8ccdedf22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 15:49:03.382883 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.382854 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2777118f-3934-4227-af12-c6d8ccdedf22-istio-token" (OuterVolumeSpecName: "istio-token") pod "2777118f-3934-4227-af12-c6d8ccdedf22" (UID: "2777118f-3934-4227-af12-c6d8ccdedf22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:49:03.382979 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.382879 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2777118f-3934-4227-af12-c6d8ccdedf22-kube-api-access-kb65j" (OuterVolumeSpecName: "kube-api-access-kb65j") pod "2777118f-3934-4227-af12-c6d8ccdedf22" (UID: "2777118f-3934-4227-af12-c6d8ccdedf22"). InnerVolumeSpecName "kube-api-access-kb65j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 15:49:03.383080 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.383066 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2777118f-3934-4227-af12-c6d8ccdedf22" (UID: "2777118f-3934-4227-af12-c6d8ccdedf22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 15:49:03.482543 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.482510 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-istio-envoy\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:49:03.482543 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.482541 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb65j\" (UniqueName: \"kubernetes.io/projected/2777118f-3934-4227-af12-c6d8ccdedf22-kube-api-access-kb65j\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:49:03.482543 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.482551 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2777118f-3934-4227-af12-c6d8ccdedf22-istio-podinfo\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:49:03.482781 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.482560 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2777118f-3934-4227-af12-c6d8ccdedf22-istio-token\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:49:03.482781 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.482567 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-istio-data\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:49:03.482781 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.482575 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2777118f-3934-4227-af12-c6d8ccdedf22-credential-socket\") on node \"ip-10-0-128-22.ec2.internal\" DevicePath \"\"" Jul 22 15:49:03.909718 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.909687 2580 generic.go:356] "Generic (PLEG): container finished" podID="2777118f-3934-4227-af12-c6d8ccdedf22" containerID="f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121" exitCode=0 Jul 22 15:49:03.909892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.909730 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" Jul 22 15:49:03.909892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.909766 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" event={"ID":"2777118f-3934-4227-af12-c6d8ccdedf22","Type":"ContainerDied","Data":"f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121"} Jul 22 15:49:03.909892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.909811 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj" event={"ID":"2777118f-3934-4227-af12-c6d8ccdedf22","Type":"ContainerDied","Data":"e3ddcd0dcb3f89036b143795f56f869d544c969f46c937e0285db4fa040e017c"} Jul 22 15:49:03.909892 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.909834 2580 scope.go:117] "RemoveContainer" containerID="f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121" Jul 22 15:49:03.919849 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.919831 2580 scope.go:117] "RemoveContainer" containerID="a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777" Jul 22 15:49:03.927330 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.927312 2580 scope.go:117] "RemoveContainer" containerID="f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121" Jul 22 15:49:03.927589 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:49:03.927571 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121\": container with ID starting with f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121 not found: ID does not exist" containerID="f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121" Jul 22 15:49:03.927642 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.927597 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121"} err="failed to get container status \"f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121\": rpc error: code = NotFound desc = could not find container \"f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121\": container with ID starting with f700f289afc9971208b60f8e3d67fb326663357cef2c9335548f074d8ca2f121 not found: ID does not exist" Jul 22 15:49:03.927642 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.927615 2580 scope.go:117] "RemoveContainer" containerID="a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777" Jul 22 15:49:03.927857 ip-10-0-128-22 kubenswrapper[2580]: E0722 15:49:03.927839 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777\": container with ID starting with a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777 not found: ID does not exist" containerID="a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777" Jul 22 15:49:03.927899 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.927862 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777"} err="failed to get container status \"a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777\": rpc error: code = NotFound desc = could not find container \"a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777\": container with ID starting with a5ab6f0632c9b8a9d36b684b82d852c537e0919c3d9da2c3ed5eaac3d2789777 not found: ID does not exist" Jul 22 15:49:03.941504 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.938641 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj"] Jul 22 15:49:03.944355 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:03.944331 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-552a8b-openshift-default-6fbc8b89c5-67knj"] Jul 22 15:49:04.693640 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:04.693609 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wrlvf_b202d9b6-1e29-4995-aa00-251fdf0b356b/manager/0.log" Jul 22 15:49:04.947844 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:04.947746 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bz9tp_f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8/server/0.log" Jul 22 15:49:05.053766 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:05.053734 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" path="/var/lib/kubelet/pods/2777118f-3934-4227-af12-c6d8ccdedf22/volumes" Jul 22 15:49:05.073153 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:05.073111 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ptwkj_0a5d6422-e617-4392-bb78-ea9f33304047/manager/0.log" Jul 22 15:49:05.336709 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:05.336679 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7777b6b677-t2j4z_0a8f4ced-8d46-49b5-bd5e-89a35811254f/manager/0.log" Jul 22 15:49:05.343091 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:05.343073 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7777b6b677-t2j4z_0a8f4ced-8d46-49b5-bd5e-89a35811254f/copy-manifests/0.log" Jul 22 15:49:05.985478 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:05.985448 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745555-nrwd7_c7f016b3-8e40-4cbe-a4f4-533a68c9bec3/cleanup/0.log" Jul 22 15:49:08.054431 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:08.054401 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-qvjnm_eae0bf69-caf9-4256-bf15-20edc3e97e57/manager/0.log" Jul 22 15:49:08.302131 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:08.302097 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zsrln_96088db1-f57f-4817-b4b0-d64031c78567/registry-server/0.log" Jul 22 15:49:08.453313 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:08.453279 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hp5rq_d27f993d-22bb-4636-ae76-cc7981d2cd72/manager/0.log" Jul 22 15:49:08.585326 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:08.585298 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-d697m_00778b00-ec3b-4341-8166-6474593daafb/limitador/0.log" Jul 22 15:49:09.234837 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:09.234810 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-nxctp_7734426c-3de3-4d5c-9bf4-ce2860bade98/discovery/0.log" Jul 22 15:49:09.494010 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:09.493938 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59659ff488-rp2tq_0fb85daf-a839-45f0-bf23-24cfc9ed2f06/kube-auth-proxy/0.log" Jul 22 15:49:09.617469 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:09.617443 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl7q9_f32c102b-7deb-4be1-a86b-a060709ae9b0/istio-proxy/1.log" Jul 22 15:49:10.575567 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:10.575524 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-snf67_d63a3b0f-12e3-454e-9348-57c7c8cff3a5/main/0.log" Jul 22 15:49:19.984419 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:19.984372 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bjpkw_137d785b-18a1-41d5-9e16-d4c7f5f7af6b/global-pull-secret-syncer/0.log" Jul 22 15:49:20.227950 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:20.227917 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b695k_24cb6d68-e498-4801-b5b9-76ae50893cc4/konnectivity-agent/0.log" Jul 22 15:49:20.249529 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:20.249450 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-22.ec2.internal_5e61fc359e18a19e5497b5e6df8484f9/haproxy/0.log" Jul 22 15:49:25.144212 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:25.144184 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-qvjnm_eae0bf69-caf9-4256-bf15-20edc3e97e57/manager/0.log" Jul 22 15:49:25.221227 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:25.221202 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zsrln_96088db1-f57f-4817-b4b0-d64031c78567/registry-server/0.log" Jul 22 15:49:25.328048 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:25.328016 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hp5rq_d27f993d-22bb-4636-ae76-cc7981d2cd72/manager/0.log" Jul 22 15:49:25.360155 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:25.360129 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-d697m_00778b00-ec3b-4341-8166-6474593daafb/limitador/0.log" Jul 22 15:49:27.637423 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:27.637395 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cbmj7_a8857aff-0139-4cbc-b7c3-9761099e4db2/node-exporter/0.log" Jul 22 15:49:27.659205 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:27.659157 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cbmj7_a8857aff-0139-4cbc-b7c3-9761099e4db2/kube-rbac-proxy/0.log" Jul 22 15:49:27.683601 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:27.683580 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cbmj7_a8857aff-0139-4cbc-b7c3-9761099e4db2/init-textfile/0.log" Jul 22 15:49:30.192799 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.192762 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv"] Jul 22 15:49:30.193204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193074 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" containerName="istio-proxy" Jul 22 15:49:30.193204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193085 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" containerName="istio-proxy" Jul 22 15:49:30.193204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193093 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47fc0362-884d-4e4b-a335-8b392f2d2697" containerName="main" Jul 22 15:49:30.193204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193099 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="47fc0362-884d-4e4b-a335-8b392f2d2697" containerName="main" Jul 22 15:49:30.193204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193108 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" Jul 22 15:49:30.193204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193129 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" Jul 22 15:49:30.193204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193135 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" containerName="istio-proxy" Jul 22 15:49:30.193204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193142 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" containerName="istio-proxy" Jul 22 15:49:30.193204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193149 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" Jul 22 15:49:30.193204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193153 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" Jul 22 15:49:30.193516 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193218 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="47fc0362-884d-4e4b-a335-8b392f2d2697" containerName="main" Jul 22 15:49:30.193516 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193227 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" containerName="istio-proxy" Jul 22 15:49:30.193516 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193232 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" Jul 22 15:49:30.193516 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193238 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79d6dd09-2686-40ff-99c8-785c5a79116c" containerName="istio-proxy" Jul 22 15:49:30.193516 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.193243 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2777118f-3934-4227-af12-c6d8ccdedf22" containerName="istio-proxy" Jul 22 15:49:30.196206 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.196186 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.199204 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.199182 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6tddk\"/\"default-dockercfg-9pgbb\"" Jul 22 15:49:30.200437 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.200419 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6tddk\"/\"kube-root-ca.crt\"" Jul 22 15:49:30.200520 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.200470 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6tddk\"/\"openshift-service-ca.crt\"" Jul 22 15:49:30.206017 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.205992 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv"] Jul 22 15:49:30.297449 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.297411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8eca64f0-7bb6-45b2-84fd-44c0606a4891-lib-modules\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.297640 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.297478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8eca64f0-7bb6-45b2-84fd-44c0606a4891-podres\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.297640 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.297512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8eca64f0-7bb6-45b2-84fd-44c0606a4891-proc\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.297640 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.297537 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-568l8\" (UniqueName: \"kubernetes.io/projected/8eca64f0-7bb6-45b2-84fd-44c0606a4891-kube-api-access-568l8\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.297640 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.297591 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8eca64f0-7bb6-45b2-84fd-44c0606a4891-sys\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.398445 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.398412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8eca64f0-7bb6-45b2-84fd-44c0606a4891-proc\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.398638 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.398452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-568l8\" (UniqueName: \"kubernetes.io/projected/8eca64f0-7bb6-45b2-84fd-44c0606a4891-kube-api-access-568l8\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.398638 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.398481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8eca64f0-7bb6-45b2-84fd-44c0606a4891-sys\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.398638 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.398500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8eca64f0-7bb6-45b2-84fd-44c0606a4891-lib-modules\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.398638 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.398542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8eca64f0-7bb6-45b2-84fd-44c0606a4891-podres\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.398638 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.398563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8eca64f0-7bb6-45b2-84fd-44c0606a4891-sys\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.398638 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.398558 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8eca64f0-7bb6-45b2-84fd-44c0606a4891-proc\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.398866 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.398649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8eca64f0-7bb6-45b2-84fd-44c0606a4891-podres\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.398866 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.398658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8eca64f0-7bb6-45b2-84fd-44c0606a4891-lib-modules\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.407650 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.407620 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-568l8\" (UniqueName: \"kubernetes.io/projected/8eca64f0-7bb6-45b2-84fd-44c0606a4891-kube-api-access-568l8\") pod \"perf-node-gather-daemonset-qfntv\" (UID: \"8eca64f0-7bb6-45b2-84fd-44c0606a4891\") " pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.507192 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.507089 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:30.640480 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:30.640452 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv"] Jul 22 15:49:30.643703 ip-10-0-128-22 kubenswrapper[2580]: W0722 15:49:30.643672 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8eca64f0_7bb6_45b2_84fd_44c0606a4891.slice/crio-e45904e77fa01c35fa940d99ff641bd1d532c85c7e75c7d6548efac04081df2a WatchSource:0}: Error finding container e45904e77fa01c35fa940d99ff641bd1d532c85c7e75c7d6548efac04081df2a: Status 404 returned error can't find the container with id e45904e77fa01c35fa940d99ff641bd1d532c85c7e75c7d6548efac04081df2a Jul 22 15:49:31.035407 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:31.035378 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" event={"ID":"8eca64f0-7bb6-45b2-84fd-44c0606a4891","Type":"ContainerStarted","Data":"218da8d3af17f88a7f27bd5a9a5b40dd9407eabacd93933fa33a7eabb8cd901f"} Jul 22 15:49:31.035577 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:31.035452 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" event={"ID":"8eca64f0-7bb6-45b2-84fd-44c0606a4891","Type":"ContainerStarted","Data":"e45904e77fa01c35fa940d99ff641bd1d532c85c7e75c7d6548efac04081df2a"} Jul 22 15:49:31.035577 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:31.035524 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:31.058628 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:31.058578 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" podStartSLOduration=1.058564199 podStartE2EDuration="1.058564199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 15:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 15:49:31.05608153 +0000 UTC m=+2696.554981306" watchObservedRunningTime="2026-07-22 15:49:31.058564199 +0000 UTC m=+2696.557463974" Jul 22 15:49:32.763391 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:32.763365 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vqrb5_ea197d01-92b2-44c6-87ac-7413a45fc26c/dns/0.log" Jul 22 15:49:32.784697 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:32.784677 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vqrb5_ea197d01-92b2-44c6-87ac-7413a45fc26c/kube-rbac-proxy/0.log" Jul 22 15:49:32.832980 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:32.832960 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t76c6_7003d3d9-bb04-433c-a3f4-a5e95f7f93da/dns-node-resolver/0.log" Jul 22 15:49:33.508965 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:33.508938 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rrrth_e661d076-cf63-4136-9ac5-3afa121bda96/node-ca/0.log" Jul 22 15:49:34.586814 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:34.586789 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-nxctp_7734426c-3de3-4d5c-9bf4-ce2860bade98/discovery/0.log" Jul 22 15:49:34.643067 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:34.643027 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59659ff488-rp2tq_0fb85daf-a839-45f0-bf23-24cfc9ed2f06/kube-auth-proxy/0.log" Jul 22 15:49:34.676154 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:34.676107 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl7q9_f32c102b-7deb-4be1-a86b-a060709ae9b0/istio-proxy/0.log" Jul 22 15:49:34.700730 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:34.700701 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl7q9_f32c102b-7deb-4be1-a86b-a060709ae9b0/istio-proxy/1.log" Jul 22 15:49:35.536190 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:35.536106 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2487l_65817246-491c-4aea-9ce0-2f582b22652f/serve-healthcheck-canary/0.log" Jul 22 15:49:36.263223 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:36.263164 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qfxjl_4f42ba2c-c6f7-49e8-92bb-4ade149a922d/kube-rbac-proxy/0.log" Jul 22 15:49:36.282434 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:36.282413 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qfxjl_4f42ba2c-c6f7-49e8-92bb-4ade149a922d/exporter/0.log" Jul 22 15:49:36.303998 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:36.303974 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qfxjl_4f42ba2c-c6f7-49e8-92bb-4ade149a922d/extractor/0.log" Jul 22 15:49:37.053620 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:37.053596 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6tddk/perf-node-gather-daemonset-qfntv" Jul 22 15:49:38.681013 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:38.680975 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wrlvf_b202d9b6-1e29-4995-aa00-251fdf0b356b/manager/0.log" Jul 22 15:49:38.922941 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:38.922912 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bz9tp_f7d40f5b-9a8a-4650-b1a2-51ba2a866fe8/server/0.log" Jul 22 15:49:38.995633 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:38.995551 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ptwkj_0a5d6422-e617-4392-bb78-ea9f33304047/manager/0.log" Jul 22 15:49:39.185368 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:39.185341 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7777b6b677-t2j4z_0a8f4ced-8d46-49b5-bd5e-89a35811254f/manager/0.log" Jul 22 15:49:39.206791 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:39.206764 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7777b6b677-t2j4z_0a8f4ced-8d46-49b5-bd5e-89a35811254f/copy-manifests/0.log" Jul 22 15:49:41.151037 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:41.151003 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-nxctp_7734426c-3de3-4d5c-9bf4-ce2860bade98/discovery/0.log" Jul 22 15:49:41.196817 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:41.196792 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59659ff488-rp2tq_0fb85daf-a839-45f0-bf23-24cfc9ed2f06/kube-auth-proxy/0.log" Jul 22 15:49:41.215896 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:41.215868 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl7q9_f32c102b-7deb-4be1-a86b-a060709ae9b0/istio-proxy/0.log" Jul 22 15:49:41.240909 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:41.240880 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl7q9_f32c102b-7deb-4be1-a86b-a060709ae9b0/istio-proxy/1.log" Jul 22 15:49:42.612655 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:42.612624 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-nxctp_7734426c-3de3-4d5c-9bf4-ce2860bade98/discovery/0.log" Jul 22 15:49:42.645291 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:42.645265 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59659ff488-rp2tq_0fb85daf-a839-45f0-bf23-24cfc9ed2f06/kube-auth-proxy/0.log" Jul 22 15:49:42.666720 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:42.666694 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl7q9_f32c102b-7deb-4be1-a86b-a060709ae9b0/istio-proxy/0.log" Jul 22 15:49:42.692697 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:42.692675 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl7q9_f32c102b-7deb-4be1-a86b-a060709ae9b0/istio-proxy/1.log" Jul 22 15:49:46.141884 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:46.141800 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-x8w7k_3c99be67-6a07-4e56-b299-4076b37cb720/migrator/0.log" Jul 22 15:49:46.170219 ip-10-0-128-22 kubenswrapper[2580]: I0722 15:49:46.170191 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-x8w7k_3c99be67-6a07-4e56-b299-4076b37cb720/graceful-termination/0.log"