Sep 17 23:04:13.786769 ip-10-0-133-222 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 17 23:04:13.786780 ip-10-0-133-222 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 17 23:04:13.786787 ip-10-0-133-222 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 17 23:04:13.786958 ip-10-0-133-222 systemd[1]: Failed to start Kubernetes Kubelet. Sep 17 23:04:23.931917 ip-10-0-133-222 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 17 23:04:23.931938 ip-10-0-133-222 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c4cb4ace139e47849518c7645a46eb81 -- Sep 17 23:06:10.236291 ip-10-0-133-222 systemd[1]: Starting Kubernetes Kubelet... Sep 17 23:06:14.052477 ip-10-0-133-222 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 23:06:14.052477 ip-10-0-133-222 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 23:06:14.052477 ip-10-0-133-222 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 23:06:14.052477 ip-10-0-133-222 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 23:06:14.052477 ip-10-0-133-222 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 23:06:14.053386 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.052545 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 23:06:14.055611 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055594 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055612 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055618 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055621 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055624 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055628 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055630 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055633 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055637 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055639 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055642 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055645 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055647 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055651 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055654 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055657 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055660 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 23:06:14.055654 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055663 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055666 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055670 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055672 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055675 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055678 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055681 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055684 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055687 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055689 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055692 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055695 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055698 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055701 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055703 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055706 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055709 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055712 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055715 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055717 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 23:06:14.056108 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055720 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055722 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055725 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055728 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055731 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055734 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055737 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055740 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055742 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055745 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055748 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055750 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055753 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055756 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055760 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055762 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055765 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055768 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055771 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 23:06:14.056666 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055776 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055780 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055783 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055786 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055789 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055792 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055795 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055798 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055800 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055803 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055806 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055809 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055811 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055814 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055817 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055820 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055823 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055828 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055831 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055834 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 23:06:14.057180 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055837 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055839 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055842 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055845 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055848 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055850 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055853 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055856 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055859 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055862 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055865 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055868 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055872 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055875 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055878 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055882 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055885 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055888 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055891 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055894 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 23:06:14.057719 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055897 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055900 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.055902 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056763 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056771 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056775 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056779 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056782 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056786 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056790 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056793 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056796 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056800 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056803 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056805 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056808 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056811 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056814 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056816 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056820 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 23:06:14.058268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056823 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056826 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056829 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056832 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056835 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056838 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056840 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056843 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056847 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056851 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056855 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056858 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056861 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056864 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056867 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056870 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056872 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056875 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056878 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 23:06:14.058802 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056881 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056883 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056887 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056890 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056893 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056896 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056900 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056903 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056907 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056910 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056912 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056915 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056920 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056923 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056926 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056930 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056933 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056935 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056938 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056941 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 23:06:14.059327 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056944 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056947 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056949 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056952 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056955 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056957 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056960 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056963 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056965 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056969 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056971 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056991 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056994 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.056997 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057000 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057003 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057006 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057009 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057012 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057016 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 23:06:14.059853 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057019 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057022 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057025 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057028 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057030 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057033 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057036 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057039 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057043 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057046 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057048 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057051 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057054 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057057 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057059 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057063 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057065 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057069 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057073 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 23:06:14.060385 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057077 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057080 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057083 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057085 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057466 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057471 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057475 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057478 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057480 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057483 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057486 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057489 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057493 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057496 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057499 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057501 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057504 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057507 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057510 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057512 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 23:06:14.060905 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057515 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057518 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057521 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057524 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057526 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057529 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057532 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057535 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057537 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057540 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057543 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057545 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057548 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057551 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057554 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057557 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057559 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057562 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057565 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057568 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 23:06:14.061445 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057571 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057574 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057576 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057579 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057582 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057585 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057588 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057590 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057593 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057596 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057615 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057619 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057622 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057625 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057628 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057631 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057633 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057636 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057639 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057642 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 23:06:14.061988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057644 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057647 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057650 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057653 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057670 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057673 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057678 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057682 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057684 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057688 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057691 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057694 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057698 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057702 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057706 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057709 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057712 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057716 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057720 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 23:06:14.062660 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057725 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057729 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057732 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057736 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057739 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057742 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057746 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057749 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057753 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057757 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057760 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057764 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057766 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057769 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057772 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057775 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057777 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057781 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057783 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057787 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 23:06:14.063195 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057790 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057793 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057796 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.057799 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057872 2559 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057905 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057911 2559 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057916 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057921 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057924 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057929 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057933 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057938 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057941 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057945 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057948 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057951 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057955 2559 flags.go:64] FLAG: --cgroup-root="" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057958 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057961 2559 flags.go:64] FLAG: --client-ca-file="" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057964 2559 flags.go:64] FLAG: --cloud-provider="external" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057967 2559 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057971 2559 flags.go:64] FLAG: --cluster-domain="" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057989 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 23:06:14.063753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057992 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057996 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.057999 2559 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058003 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058007 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058010 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058014 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058019 2559 flags.go:64] FLAG: --contention-profiling="false" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058022 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058026 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058030 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058033 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058037 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058040 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058043 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058047 2559 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058050 2559 flags.go:64] FLAG: --enable-server="true" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058053 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058058 2559 flags.go:64] FLAG: --event-burst="100" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058062 2559 flags.go:64] FLAG: --event-qps="50" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058065 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058069 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058072 2559 flags.go:64] FLAG: --eviction-hard="" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058076 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 23:06:14.064450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058079 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058083 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058086 2559 flags.go:64] FLAG: --eviction-soft="" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058090 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058093 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058096 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058099 2559 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058102 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058105 2559 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058108 2559 flags.go:64] FLAG: --feature-gates="" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058112 2559 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058115 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058119 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058122 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058128 2559 flags.go:64] FLAG: --healthz-port="10248" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058132 2559 flags.go:64] FLAG: --help="false" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058136 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058139 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058143 2559 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058146 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058150 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058153 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058156 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 23:06:14.065088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058160 2559 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058162 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058166 2559 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058169 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058172 2559 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058175 2559 flags.go:64] FLAG: --kube-reserved="" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058178 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058182 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058185 2559 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058188 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058191 2559 flags.go:64] FLAG: --lock-file="" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058194 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058197 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058200 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058206 2559 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058209 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058212 2559 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058215 2559 flags.go:64] FLAG: --logging-format="text" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058218 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058222 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058225 2559 flags.go:64] FLAG: --manifest-url="" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058228 2559 flags.go:64] FLAG: --manifest-url-header="" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058232 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058238 2559 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058244 2559 flags.go:64] FLAG: --max-pods="110" Sep 17 23:06:14.065954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058247 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058251 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058254 2559 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058257 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058260 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058263 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058266 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058274 2559 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058278 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058281 2559 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058285 2559 flags.go:64] FLAG: --pod-cidr="" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058288 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058294 2559 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058297 2559 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058300 2559 flags.go:64] FLAG: --pods-per-core="0" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058304 2559 flags.go:64] FLAG: --port="10250" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058308 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058311 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fc790c426888c4a6" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058314 2559 flags.go:64] FLAG: --qos-reserved="" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058317 2559 flags.go:64] FLAG: --read-only-port="10255" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058320 2559 flags.go:64] FLAG: --register-node="true" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058323 2559 flags.go:64] FLAG: --register-with-taints="" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058327 2559 flags.go:64] FLAG: --registry-burst="10" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058330 2559 flags.go:64] FLAG: --registry-qps="5" Sep 17 23:06:14.067169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058333 2559 flags.go:64] FLAG: --reserved-cpus="" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058336 2559 flags.go:64] FLAG: --reserved-memory="" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058340 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058343 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058346 2559 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058349 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058352 2559 flags.go:64] FLAG: --runonce="false" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058360 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058363 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058366 2559 flags.go:64] FLAG: --seccomp-default="false" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058369 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058373 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058376 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058379 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058383 2559 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058386 2559 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058389 2559 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058392 2559 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058395 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058398 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058401 2559 flags.go:64] FLAG: --system-cgroups="" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058404 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058410 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058413 2559 flags.go:64] FLAG: --tls-cert-file="" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058418 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 23:06:14.068296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058421 2559 flags.go:64] FLAG: --tls-min-version="" Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058424 2559 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058427 2559 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058430 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058433 2559 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058436 2559 flags.go:64] FLAG: --v="2" Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058440 2559 flags.go:64] FLAG: --version="false" Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058445 2559 flags.go:64] FLAG: --vmodule="" Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058448 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.058452 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058592 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058596 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058599 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058603 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058609 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058612 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058615 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058619 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058623 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058626 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058629 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 23:06:14.069506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058632 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058635 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058638 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058641 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058644 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058647 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058650 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058652 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058655 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058658 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058661 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058668 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058671 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058673 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058676 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058679 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058682 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058685 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058688 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058691 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 23:06:14.070521 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058694 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058698 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058702 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058705 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058709 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058718 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058722 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058724 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058728 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058731 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058734 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058738 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058741 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058744 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058747 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058749 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058752 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058755 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058758 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 23:06:14.071258 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058761 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058764 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058767 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058769 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058772 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058775 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058778 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058781 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058783 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058786 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058789 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058792 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058795 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058797 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058800 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058803 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058806 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058809 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058815 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058818 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 23:06:14.072145 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058821 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058824 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058826 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058829 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058832 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058835 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058838 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058842 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058846 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058849 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058852 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058855 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058858 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058860 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058863 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058866 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058869 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058871 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058874 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058877 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 17 23:06:14.072946 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058880 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058882 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058885 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058888 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058891 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058893 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058896 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058899 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.058902 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.059444 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.068609 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.068631 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.068662 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.068676 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068785 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 23:06:14.073518 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068793 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068799 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068804 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068809 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068814 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068820 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068825 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068830 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068836 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068840 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068845 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068849 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068854 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068858 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068863 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068867 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068872 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068876 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068881 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068886 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 17 23:06:14.073995 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068891 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068895 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068900 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068904 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068909 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068913 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068918 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068923 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068937 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068942 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068947 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068951 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068956 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068960 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068965 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068969 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068993 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.068998 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069003 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069007 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 23:06:14.074653 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069011 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069016 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069021 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069026 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069033 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069039 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069044 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069048 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069053 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069058 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069062 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069068 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069073 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069078 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069086 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069091 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069096 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069100 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 23:06:14.075373 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069105 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069110 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069116 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069130 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069135 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069139 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069144 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069148 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069153 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069158 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069162 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069167 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069171 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069176 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069181 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069185 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069189 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069196 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069202 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 23:06:14.076151 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069207 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069212 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069216 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069221 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069226 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069230 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069235 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069239 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069244 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069248 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069253 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069258 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069262 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069266 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069270 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069276 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069280 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069298 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069304 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069309 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 23:06:14.076773 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069314 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.069324 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069610 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069621 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069626 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069632 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069637 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069642 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069647 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069651 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069656 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069660 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069665 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069670 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069675 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 23:06:14.077434 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069679 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069683 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069688 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069693 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069698 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069702 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069707 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069712 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069717 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069721 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069725 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069730 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069734 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069739 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069744 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069760 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069767 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069772 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069777 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069783 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 23:06:14.077833 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069788 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069793 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069799 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069804 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069809 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069814 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069819 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069823 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069827 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069832 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069837 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069842 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069847 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069851 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069858 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069864 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069869 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069874 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069879 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 23:06:14.078380 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069884 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069888 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069893 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069897 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069902 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069906 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069910 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069915 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069920 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069934 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069939 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069943 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069948 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069952 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069957 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069961 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069966 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069971 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.069998 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070003 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 23:06:14.078880 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070020 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070025 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070029 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070034 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070039 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070043 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070047 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070051 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070057 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070062 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070066 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070070 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070075 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070079 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070084 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070090 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070096 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070101 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070105 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070110 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 23:06:14.079430 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070114 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070119 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070132 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070136 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070141 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070145 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:14.070150 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.070158 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.070379 2559 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.073958 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.075633 2559 server.go:1019] "Starting client certificate rotation" Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.075921 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.075985 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 23:06:14.079950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.079132 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 23:06:14.080347 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.080330 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 23:06:14.093090 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.093071 2559 log.go:25] "Validated CRI v1 runtime API" Sep 17 23:06:14.094715 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.094702 2559 log.go:25] "Validated CRI v1 image API" Sep 17 23:06:14.095637 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.095616 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 23:06:14.099342 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.099315 2559 fs.go:135] Filesystem UUIDs: map[2cce2163-252c-4cb1-ad87-3c01e0e880de:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9cc04373-7389-4dd0-a7a0-b23e541b3a37:/dev/nvme0n1p3] Sep 17 23:06:14.099342 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.099334 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 23:06:14.105039 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.104913 2559 manager.go:216] Machine: {Timestamp:2026-09-17 23:06:14.104421278 +0000 UTC m=+2.221900744 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092186 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26952d8fa46e37d5c53fb68587a4ca SystemUUID:ec26952d-8fa4-6e37-d5c5-3fb68587a4ca BootID:c4cb4ace-139e-4784-9518-c7645a46eb81 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:54:ff:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:54:ff:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:ac:36:26:87:fb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 23:06:14.105039 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.105029 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 23:06:14.105172 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.105109 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 23:06:14.106892 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.106866 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 23:06:14.107057 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.106895 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 23:06:14.108323 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.108312 2559 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 23:06:14.108360 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.108326 2559 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 23:06:14.108360 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.108334 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 23:06:14.108360 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.108346 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 23:06:14.108446 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.108417 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 23:06:14.111574 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.111560 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 17 23:06:14.111692 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.111682 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 23:06:14.111743 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.111716 2559 kubelet.go:486] "Attempting to sync node with API server" Sep 17 23:06:14.111743 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.111727 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 23:06:14.111743 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.111741 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 23:06:14.111855 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.111751 2559 kubelet.go:398] "Adding apiserver pod source" Sep 17 23:06:14.111855 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.111760 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 23:06:14.112196 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.112184 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 23:06:14.112238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.112205 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 23:06:14.119565 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.119538 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 23:06:14.120151 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.120087 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 17 23:06:14.123880 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.123861 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 23:06:14.123956 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.123895 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 23:06:14.125132 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125118 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 23:06:14.125205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125137 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 23:06:14.125205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125144 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 23:06:14.125205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125149 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 23:06:14.125205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125161 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 23:06:14.125205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125169 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 23:06:14.125205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125184 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 23:06:14.125205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125189 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 23:06:14.125205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125196 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 23:06:14.125205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125202 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 23:06:14.125205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125211 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 23:06:14.125485 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125217 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 23:06:14.125485 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125242 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 23:06:14.125485 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.125248 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 23:06:14.125641 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.125616 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 23:06:14.125685 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.125615 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 23:06:14.129123 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.129109 2559 server.go:1268] "Started kubelet" Sep 17 23:06:14.129221 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.129167 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 23:06:14.129262 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.129200 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 23:06:14.129293 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.129272 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 23:06:14.130022 ip-10-0-133-222 systemd[1]: Started Kubernetes Kubelet. Sep 17 23:06:14.130599 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.130582 2559 server.go:310] "Adding debug handlers to kubelet server" Sep 17 23:06:14.130904 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.130832 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 23:06:14.135884 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.135865 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 23:06:14.136126 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.136113 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-222.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.138209 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.138237 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.138375 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.139035 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.139052 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.139195 2559 reconstruct.go:97] "Volume reconstruction finished" Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.139204 2559 reconciler.go:29] "Reconciler: start to sync state" Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.139391 2559 factory.go:55] Registering systemd factory Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.139405 2559 factory.go:223] Registration of the systemd container factory successfully Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.139407 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.139659 2559 factory.go:153] Registering CRI-O factory Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.139669 2559 factory.go:223] Registration of the crio container factory successfully Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.139735 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.139750 2559 factory.go:103] Registering Raw factory Sep 17 23:06:14.140249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.139765 2559 manager.go:1196] Started watching for new ooms in manager Sep 17 23:06:14.141004 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.140835 2559 manager.go:349] Starting recovery of all containers Sep 17 23:06:14.149214 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.149186 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 23:06:14.149317 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.149296 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 23:06:14.152574 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.152558 2559 manager.go:354] Recovery completed Sep 17 23:06:14.166661 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.166641 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5vkwp" Sep 17 23:06:14.168655 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.168641 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:14.171217 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.171201 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:14.171326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.171234 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:14.171326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.171250 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:14.171790 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.171768 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 23:06:14.171790 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.171786 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 23:06:14.171790 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.171789 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:14.171965 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.171806 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 17 23:06:14.172556 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.172539 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:14.172648 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.172571 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:14.172648 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.172583 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:14.173062 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.173045 2559 policy_none.go:49] "None policy: Start" Sep 17 23:06:14.173062 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.173065 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 23:06:14.173182 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.173079 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 23:06:14.174114 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.174102 2559 policy_none.go:47] "Start" Sep 17 23:06:14.174521 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.174496 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5vkwp" Sep 17 23:06:14.176056 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.175013 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-222.ec2.internal.18d63e466d521f1a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-222.ec2.internal,UID:ip-10-0-133-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-222.ec2.internal,},FirstTimestamp:2026-09-17 23:06:14.129082138 +0000 UTC m=+2.246561583,LastTimestamp:2026-09-17 23:06:14.129082138 +0000 UTC m=+2.246561583,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-222.ec2.internal,}" Sep 17 23:06:14.215484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.208554 2559 manager.go:337] "Starting Device Plugin manager" Sep 17 23:06:14.215484 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.208587 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 23:06:14.215484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.208597 2559 server.go:87] "Starting device plugin registration server" Sep 17 23:06:14.215484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.208848 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 23:06:14.215484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.208874 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 23:06:14.215484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.208984 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 23:06:14.215484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.209100 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 23:06:14.215484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.209114 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 23:06:14.215484 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.209605 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 23:06:14.215484 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.209646 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-222.ec2.internal\" not found" Sep 17 23:06:14.278860 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.278831 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 23:06:14.280175 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.280151 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 23:06:14.280274 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.280200 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 23:06:14.280274 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.280247 2559 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 23:06:14.280373 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.280285 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 23:06:14.282833 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.282817 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:14.309798 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.309753 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:14.310592 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.310574 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:14.310684 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.310609 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:14.310684 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.310625 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:14.310684 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.310654 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.320568 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.320550 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.320641 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.320571 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-222.ec2.internal\": node \"ip-10-0-133-222.ec2.internal\" not found" Sep 17 23:06:14.355038 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.355018 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Sep 17 23:06:14.380892 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.380856 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal"] Sep 17 23:06:14.381006 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.380974 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:14.381822 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.381786 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:14.381822 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.381824 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:14.382009 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.381839 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:14.383428 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.383412 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:14.383550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.383534 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.383601 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.383571 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:14.384168 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.384147 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:14.384253 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.384179 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:14.384253 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.384193 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:14.384253 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.384149 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:14.384253 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.384236 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:14.384253 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.384246 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:14.385473 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.385457 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.385565 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.385482 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:14.386188 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.386173 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:14.386268 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.386197 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:14.386268 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.386207 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:14.406661 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.406638 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-222.ec2.internal\" not found" node="ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.411196 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.411181 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-222.ec2.internal\" not found" node="ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.440806 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.440783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ac9bbdb0f7baee4c4f7c0e8cb2517fc-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.ec2.internal\" (UID: \"9ac9bbdb0f7baee4c4f7c0e8cb2517fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.440880 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.440809 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a4ebc53fbb6af3fc3b4be0312595c4c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"a4ebc53fbb6af3fc3b4be0312595c4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.440880 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.440828 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4ebc53fbb6af3fc3b4be0312595c4c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"a4ebc53fbb6af3fc3b4be0312595c4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.456158 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.456135 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Sep 17 23:06:14.541084 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.541052 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a4ebc53fbb6af3fc3b4be0312595c4c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"a4ebc53fbb6af3fc3b4be0312595c4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.541084 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.541090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4ebc53fbb6af3fc3b4be0312595c4c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"a4ebc53fbb6af3fc3b4be0312595c4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.541209 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.541107 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ac9bbdb0f7baee4c4f7c0e8cb2517fc-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.ec2.internal\" (UID: \"9ac9bbdb0f7baee4c4f7c0e8cb2517fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.541209 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.541156 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4ebc53fbb6af3fc3b4be0312595c4c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"a4ebc53fbb6af3fc3b4be0312595c4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.541209 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.541158 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a4ebc53fbb6af3fc3b4be0312595c4c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"a4ebc53fbb6af3fc3b4be0312595c4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.541209 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.541190 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ac9bbdb0f7baee4c4f7c0e8cb2517fc-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.ec2.internal\" (UID: \"9ac9bbdb0f7baee4c4f7c0e8cb2517fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.557374 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.557350 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Sep 17 23:06:14.658429 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.658370 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Sep 17 23:06:14.711616 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.711587 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.714968 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:14.714953 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Sep 17 23:06:14.759369 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.759218 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Sep 17 23:06:14.860121 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.860095 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Sep 17 23:06:14.961177 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:14.961125 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Sep 17 23:06:15.045178 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.045157 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:15.068297 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.068275 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:15.074424 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.074405 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 23:06:15.074522 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.074506 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 23:06:15.074562 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.074546 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 23:06:15.074600 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.074549 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 23:06:15.112356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.112329 2559 apiserver.go:52] "Watching apiserver" Sep 17 23:06:15.130032 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.130009 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 23:06:15.130351 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.130331 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-sq8mp","kube-system/global-pull-secret-syncer-92l5f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m","openshift-cluster-node-tuning-operator/tuned-swdt6","openshift-multus/multus-nvk6c","openshift-multus/network-metrics-daemon-vk29c","openshift-network-operator/iptables-alerter-mqng4","openshift-ovn-kubernetes/ovnkube-node-ljnfz","kube-system/konnectivity-agent-9vswh","openshift-dns/node-resolver-fxvl4","openshift-image-registry/node-ca-l5s9l","openshift-multus/multus-additional-cni-plugins-lx2wl"] Sep 17 23:06:15.131686 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.131670 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:15.131751 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.131733 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:15.132732 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.132717 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:15.132816 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.132783 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:15.134070 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.134048 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.136183 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.136157 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.136722 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.136699 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hskzb\"" Sep 17 23:06:15.137354 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.137316 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 23:06:15.137550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.137528 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 23:06:15.137993 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.137974 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.138387 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.138322 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 23:06:15.138387 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.138361 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Sep 17 23:06:15.140258 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.140232 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:15.140346 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.140307 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:15.141825 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.141807 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mqng4" Sep 17 23:06:15.142355 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.142336 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 23:06:15.142455 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.142360 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 23:06:15.142509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.142455 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 23:06:15.143145 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.143130 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k9brr\"" Sep 17 23:06:15.143240 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.143226 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.143376 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.143356 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 23:06:15.144452 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.144434 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9vswh" Sep 17 23:06:15.145829 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.145812 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fxvl4" Sep 17 23:06:15.147252 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147232 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l5s9l" Sep 17 23:06:15.147353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147264 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-sysctl-conf\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.147353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147284 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-multus-cni-dir\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.147353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147300 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-var-lib-cni-bin\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.147353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147320 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-multus-daemon-config\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.147353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147343 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.147623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147367 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-kubernetes\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.147623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147392 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e91666ad-1380-411e-8615-bdf527db6ef3-etc-tuned\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.147623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147416 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-system-cni-dir\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.147623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147440 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-cnibin\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.147623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147463 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2vv\" (UniqueName: \"kubernetes.io/projected/ea1ba703-ed87-422b-a256-bc96859795af-kube-api-access-dr2vv\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:15.147623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147504 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:15.147623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147528 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-run-k8s-cni-cncf-io\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.147623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147551 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-run-netns\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.147623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-var-lib-cni-multus\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147632 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-var-lib-kubelet\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147657 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4512e542-b8f7-4833-b2a5-9812a90f43c6-kubelet-config\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147674 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-socket-dir\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147690 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59nnm\" (UniqueName: \"kubernetes.io/projected/373402ca-a06b-4911-8c75-30d18162514a-kube-api-access-59nnm\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147712 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-sysconfig\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147735 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-etc-kubernetes\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147755 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147771 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147785 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-lib-modules\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147828 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-var-lib-kubelet\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147850 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-host\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147864 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpb97\" (UniqueName: \"kubernetes.io/projected/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-kube-api-access-kpb97\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147880 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlff5\" (UniqueName: \"kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5\") pod \"network-check-target-sq8mp\" (UID: \"13108b6b-4ed6-4ca0-893a-fd49f1ae5b78\") " pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147895 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-registration-dir\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-modprobe-d\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.148071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147967 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-sysctl-d\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.147984 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-run\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148003 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-sys\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148016 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e91666ad-1380-411e-8615-bdf527db6ef3-tmp\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148054 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxl9p\" (UniqueName: \"kubernetes.io/projected/e91666ad-1380-411e-8615-bdf527db6ef3-kube-api-access-cxl9p\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148087 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-run-multus-certs\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148115 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-systemd\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148150 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-cni-binary-copy\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148178 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-multus-socket-dir-parent\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148204 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4512e542-b8f7-4833-b2a5-9812a90f43c6-dbus\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148229 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-hostroot\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148253 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-multus-conf-dir\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148279 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-device-dir\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.148623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148321 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-sys-fs\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.149064 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.148741 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.158803 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.158784 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 23:06:15.158803 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.158801 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 23:06:15.158941 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.158801 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9wxxw\"" Sep 17 23:06:15.158941 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.158879 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 23:06:15.160282 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.160265 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 23:06:15.160282 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.160278 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jhjbj\"" Sep 17 23:06:15.160786 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.160773 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 23:06:15.160840 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.160817 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 23:06:15.160894 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.160867 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 23:06:15.162480 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.162458 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 23:06:15.162480 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.162469 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 23:06:15.162480 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.162490 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wcj72\"" Sep 17 23:06:15.162721 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.162508 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 23:06:15.162721 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.162492 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lcv7t\"" Sep 17 23:06:15.162797 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.162747 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 23:06:15.162797 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.162762 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 23:06:15.173064 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.172956 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 23:06:15.173064 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.172985 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 23:06:15.173064 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.172961 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 23:06:15.173064 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.173025 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 23:06:15.173272 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.173123 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 23:06:15.173330 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.173314 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-frpq6\"" Sep 17 23:06:15.173364 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.173345 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 23:06:15.173445 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.173428 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 23:06:15.174115 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.174095 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h67vq\"" Sep 17 23:06:15.174263 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.174237 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 23:06:15.174346 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.174283 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 23:06:15.174496 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.174478 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 23:06:15.174590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.174518 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ppcd7\"" Sep 17 23:06:15.174590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.174574 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Sep 17 23:06:15.174941 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.174911 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal"] Sep 17 23:06:15.179353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.179330 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:01:14 +0000 UTC" deadline="2028-05-15 08:33:38.95494858 +0000 UTC" Sep 17 23:06:15.179430 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.179354 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14529h27m23.775598538s" Sep 17 23:06:15.182241 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.182221 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 23:06:15.192710 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.192691 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 23:06:15.192923 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.192907 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal"] Sep 17 23:06:15.203387 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.203364 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4hrfx" Sep 17 23:06:15.211911 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.211863 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4hrfx" Sep 17 23:06:15.240869 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.240845 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 23:06:15.248679 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248659 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-multus-conf-dir\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.248751 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-device-dir\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.248751 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248734 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cd6491b-4078-4652-98cc-31a2b75c6911-cni-binary-copy\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.248751 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248745 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-multus-conf-dir\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.248862 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248781 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-log-socket\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.248862 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248809 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-cni-bin\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.248862 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248813 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-device-dir\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.248862 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248835 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-sysctl-conf\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.248862 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248858 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-multus-cni-dir\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.249093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-multus-daemon-config\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.249093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248910 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.249093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248957 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-run-openvswitch\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.249093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248980 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-sysctl-conf\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.249093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.248982 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e864e8ef-600f-4d42-8202-bd16f493e302-env-overrides\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.249093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249017 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5w65\" (UniqueName: \"kubernetes.io/projected/f9ef7c98-636c-443d-bb38-bdd4d0ab3a58-kube-api-access-q5w65\") pod \"iptables-alerter-mqng4\" (UID: \"f9ef7c98-636c-443d-bb38-bdd4d0ab3a58\") " pod="openshift-network-operator/iptables-alerter-mqng4" Sep 17 23:06:15.249093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249032 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-multus-cni-dir\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.249093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249036 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.249093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-kubernetes\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.249093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249074 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e91666ad-1380-411e-8615-bdf527db6ef3-etc-tuned\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.249093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-cnibin\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249101 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-kubernetes\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249141 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cd6491b-4078-4652-98cc-31a2b75c6911-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.249172 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249186 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-cnibin\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249172 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-run-k8s-cni-cncf-io\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-run-k8s-cni-cncf-io\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.249252 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret podName:4512e542-b8f7-4833-b2a5-9812a90f43c6 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:15.749213259 +0000 UTC m=+3.866692715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret") pod "global-pull-secret-syncer-92l5f" (UID: "4512e542-b8f7-4833-b2a5-9812a90f43c6") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-run-netns\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249308 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4512e542-b8f7-4833-b2a5-9812a90f43c6-kubelet-config\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249332 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59nnm\" (UniqueName: \"kubernetes.io/projected/373402ca-a06b-4911-8c75-30d18162514a-kube-api-access-59nnm\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-run-netns\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249364 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4512e542-b8f7-4833-b2a5-9812a90f43c6-kubelet-config\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249389 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249437 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-multus-daemon-config\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.249550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249468 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-sysconfig\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249514 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-sysconfig\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249514 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249569 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cd6491b-4078-4652-98cc-31a2b75c6911-cnibin\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249607 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e864e8ef-600f-4d42-8202-bd16f493e302-ovnkube-config\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.249576 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249639 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e864e8ef-600f-4d42-8202-bd16f493e302-ovn-node-metrics-cert\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.249665 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs podName:ea1ba703-ed87-422b-a256-bc96859795af nodeName:}" failed. No retries permitted until 2026-09-17 23:06:15.749651515 +0000 UTC m=+3.867130950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs") pod "network-metrics-daemon-vk29c" (UID: "ea1ba703-ed87-422b-a256-bc96859795af") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249701 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9ef7c98-636c-443d-bb38-bdd4d0ab3a58-host-slash\") pod \"iptables-alerter-mqng4\" (UID: \"f9ef7c98-636c-443d-bb38-bdd4d0ab3a58\") " pod="openshift-network-operator/iptables-alerter-mqng4" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249741 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66d8dd77-971b-4a83-ab58-b49b09d40f7b-konnectivity-ca\") pod \"konnectivity-agent-9vswh\" (UID: \"66d8dd77-971b-4a83-ab58-b49b09d40f7b\") " pod="kube-system/konnectivity-agent-9vswh" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249770 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-lib-modules\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249789 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-var-lib-kubelet\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpb97\" (UniqueName: \"kubernetes.io/projected/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-kube-api-access-kpb97\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249820 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6070593-232f-4a38-b8e2-3380c4abe49e-hosts-file\") pod \"node-resolver-fxvl4\" (UID: \"d6070593-232f-4a38-b8e2-3380c4abe49e\") " pod="openshift-dns/node-resolver-fxvl4" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249835 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cd6491b-4078-4652-98cc-31a2b75c6911-system-cni-dir\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249853 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-cni-netd\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-sysctl-d\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.250238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249878 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-var-lib-kubelet\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-run\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249946 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-lib-modules\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249962 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-run-multus-certs\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249987 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-sysctl-d\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249989 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crjvt\" (UniqueName: \"kubernetes.io/projected/064ea6f5-bd51-42ec-825e-25f7ea4a6723-kube-api-access-crjvt\") pod \"node-ca-l5s9l\" (UID: \"064ea6f5-bd51-42ec-825e-25f7ea4a6723\") " pod="openshift-image-registry/node-ca-l5s9l" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.249995 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-run\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250024 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-systemd\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250032 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-run-multus-certs\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-multus-socket-dir-parent\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250061 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-systemd\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250073 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cd6491b-4078-4652-98cc-31a2b75c6911-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-hostroot\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250110 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-multus-socket-dir-parent\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-sys-fs\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250148 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/064ea6f5-bd51-42ec-825e-25f7ea4a6723-host\") pod \"node-ca-l5s9l\" (UID: \"064ea6f5-bd51-42ec-825e-25f7ea4a6723\") " pod="openshift-image-registry/node-ca-l5s9l" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250167 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-hostroot\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250176 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-var-lib-openvswitch\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.250788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250179 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-sys-fs\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250195 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-run-ovn\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250217 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-var-lib-cni-bin\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250235 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-systemd-units\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250270 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-var-lib-cni-bin\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250279 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-run-netns\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250297 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-run-systemd\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250318 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctjwn\" (UniqueName: \"kubernetes.io/projected/e864e8ef-600f-4d42-8202-bd16f493e302-kube-api-access-ctjwn\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250338 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-system-cni-dir\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250366 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2vv\" (UniqueName: \"kubernetes.io/projected/ea1ba703-ed87-422b-a256-bc96859795af-kube-api-access-dr2vv\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250385 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cd6491b-4078-4652-98cc-31a2b75c6911-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250406 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66d8dd77-971b-4a83-ab58-b49b09d40f7b-agent-certs\") pod \"konnectivity-agent-9vswh\" (UID: \"66d8dd77-971b-4a83-ab58-b49b09d40f7b\") " pod="kube-system/konnectivity-agent-9vswh" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250430 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-var-lib-cni-multus\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250439 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-system-cni-dir\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250453 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-var-lib-kubelet\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250475 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-socket-dir\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250485 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-var-lib-cni-multus\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.251483 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250504 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd8k9\" (UniqueName: \"kubernetes.io/projected/d6070593-232f-4a38-b8e2-3380c4abe49e-kube-api-access-gd8k9\") pod \"node-resolver-fxvl4\" (UID: \"d6070593-232f-4a38-b8e2-3380c4abe49e\") " pod="openshift-dns/node-resolver-fxvl4" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250528 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-host-var-lib-kubelet\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250531 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-etc-openvswitch\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250591 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-etc-kubernetes\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-etc-kubernetes\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250658 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-kubelet\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250693 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-host\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlff5\" (UniqueName: \"kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5\") pod \"network-check-target-sq8mp\" (UID: \"13108b6b-4ed6-4ca0-893a-fd49f1ae5b78\") " pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250743 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-registration-dir\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250768 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6070593-232f-4a38-b8e2-3380c4abe49e-tmp-dir\") pod \"node-resolver-fxvl4\" (UID: \"d6070593-232f-4a38-b8e2-3380c4abe49e\") " pod="openshift-dns/node-resolver-fxvl4" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250784 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/064ea6f5-bd51-42ec-825e-25f7ea4a6723-serviceca\") pod \"node-ca-l5s9l\" (UID: \"064ea6f5-bd51-42ec-825e-25f7ea4a6723\") " pod="openshift-image-registry/node-ca-l5s9l" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250794 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250806 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcd75\" (UniqueName: \"kubernetes.io/projected/4cd6491b-4078-4652-98cc-31a2b75c6911-kube-api-access-lcd75\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250839 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-host\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-slash\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.252028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250849 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-registration-dir\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250878 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-modprobe-d\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250935 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-sys\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.250984 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-sys\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.251002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e91666ad-1380-411e-8615-bdf527db6ef3-tmp\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.251026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxl9p\" (UniqueName: \"kubernetes.io/projected/e91666ad-1380-411e-8615-bdf527db6ef3-kube-api-access-cxl9p\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.251049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e864e8ef-600f-4d42-8202-bd16f493e302-ovnkube-script-lib\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.251045 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e91666ad-1380-411e-8615-bdf527db6ef3-etc-modprobe-d\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.251085 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-cni-binary-copy\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.251101 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4512e542-b8f7-4833-b2a5-9812a90f43c6-dbus\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.251104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/373402ca-a06b-4911-8c75-30d18162514a-socket-dir\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.251116 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-node-log\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.251143 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9ef7c98-636c-443d-bb38-bdd4d0ab3a58-iptables-alerter-script\") pod \"iptables-alerter-mqng4\" (UID: \"f9ef7c98-636c-443d-bb38-bdd4d0ab3a58\") " pod="openshift-network-operator/iptables-alerter-mqng4" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.251184 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4512e542-b8f7-4833-b2a5-9812a90f43c6-dbus\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:15.252584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.251469 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-cni-binary-copy\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.252991 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.252720 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e91666ad-1380-411e-8615-bdf527db6ef3-etc-tuned\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.252991 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.252920 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e91666ad-1380-411e-8615-bdf527db6ef3-tmp\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.256291 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.256271 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:15.256384 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.256295 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:15.256384 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.256307 2559 projected.go:196] Error preparing data for projected volume kube-api-access-xlff5 for pod openshift-network-diagnostics/network-check-target-sq8mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:15.256384 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.256367 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5 podName:13108b6b-4ed6-4ca0-893a-fd49f1ae5b78 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:15.756350568 +0000 UTC m=+3.873830215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xlff5" (UniqueName: "kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5") pod "network-check-target-sq8mp" (UID: "13108b6b-4ed6-4ca0-893a-fd49f1ae5b78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:15.258012 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.257980 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59nnm\" (UniqueName: \"kubernetes.io/projected/373402ca-a06b-4911-8c75-30d18162514a-kube-api-access-59nnm\") pod \"aws-ebs-csi-driver-node-r4q5m\" (UID: \"373402ca-a06b-4911-8c75-30d18162514a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.258208 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.258186 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2vv\" (UniqueName: \"kubernetes.io/projected/ea1ba703-ed87-422b-a256-bc96859795af-kube-api-access-dr2vv\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:15.258270 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.258197 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxl9p\" (UniqueName: \"kubernetes.io/projected/e91666ad-1380-411e-8615-bdf527db6ef3-kube-api-access-cxl9p\") pod \"tuned-swdt6\" (UID: \"e91666ad-1380-411e-8615-bdf527db6ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.258545 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.258527 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpb97\" (UniqueName: \"kubernetes.io/projected/99ae3825-31b7-4c22-bf96-d6ee2c98f2c9-kube-api-access-kpb97\") pod \"multus-nvk6c\" (UID: \"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9\") " pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.351877 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.351847 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-var-lib-openvswitch\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.351877 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.351880 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-run-ovn\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.351908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-systemd-units\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.351923 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-run-netns\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.351946 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-run-systemd\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.351963 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctjwn\" (UniqueName: \"kubernetes.io/projected/e864e8ef-600f-4d42-8202-bd16f493e302-kube-api-access-ctjwn\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.351962 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-var-lib-openvswitch\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.351966 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-run-ovn\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.351978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cd6491b-4078-4652-98cc-31a2b75c6911-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.351977 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-systemd-units\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352006 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-run-netns\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352035 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66d8dd77-971b-4a83-ab58-b49b09d40f7b-agent-certs\") pod \"konnectivity-agent-9vswh\" (UID: \"66d8dd77-971b-4a83-ab58-b49b09d40f7b\") " pod="kube-system/konnectivity-agent-9vswh" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd8k9\" (UniqueName: \"kubernetes.io/projected/d6070593-232f-4a38-b8e2-3380c4abe49e-kube-api-access-gd8k9\") pod \"node-resolver-fxvl4\" (UID: \"d6070593-232f-4a38-b8e2-3380c4abe49e\") " pod="openshift-dns/node-resolver-fxvl4" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352098 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-run-systemd\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-etc-openvswitch\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352143 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352139 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4cd6491b-4078-4652-98cc-31a2b75c6911-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352172 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-kubelet\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352201 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-etc-openvswitch\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352222 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6070593-232f-4a38-b8e2-3380c4abe49e-tmp-dir\") pod \"node-resolver-fxvl4\" (UID: \"d6070593-232f-4a38-b8e2-3380c4abe49e\") " pod="openshift-dns/node-resolver-fxvl4" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352264 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-kubelet\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352382 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/064ea6f5-bd51-42ec-825e-25f7ea4a6723-serviceca\") pod \"node-ca-l5s9l\" (UID: \"064ea6f5-bd51-42ec-825e-25f7ea4a6723\") " pod="openshift-image-registry/node-ca-l5s9l" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352419 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcd75\" (UniqueName: \"kubernetes.io/projected/4cd6491b-4078-4652-98cc-31a2b75c6911-kube-api-access-lcd75\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352435 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-slash\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352454 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e864e8ef-600f-4d42-8202-bd16f493e302-ovnkube-script-lib\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352473 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-node-log\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352492 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9ef7c98-636c-443d-bb38-bdd4d0ab3a58-iptables-alerter-script\") pod \"iptables-alerter-mqng4\" (UID: \"f9ef7c98-636c-443d-bb38-bdd4d0ab3a58\") " pod="openshift-network-operator/iptables-alerter-mqng4" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352515 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cd6491b-4078-4652-98cc-31a2b75c6911-cni-binary-copy\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352529 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-slash\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352552 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-log-socket\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.352747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352589 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6070593-232f-4a38-b8e2-3380c4abe49e-tmp-dir\") pod \"node-resolver-fxvl4\" (UID: \"d6070593-232f-4a38-b8e2-3380c4abe49e\") " pod="openshift-dns/node-resolver-fxvl4" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352578 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-cni-bin\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352644 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-cni-bin\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352685 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-run-openvswitch\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-node-log\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352742 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-log-socket\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-run-openvswitch\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352764 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e864e8ef-600f-4d42-8202-bd16f493e302-env-overrides\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352796 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5w65\" (UniqueName: \"kubernetes.io/projected/f9ef7c98-636c-443d-bb38-bdd4d0ab3a58-kube-api-access-q5w65\") pod \"iptables-alerter-mqng4\" (UID: \"f9ef7c98-636c-443d-bb38-bdd4d0ab3a58\") " pod="openshift-network-operator/iptables-alerter-mqng4" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cd6491b-4078-4652-98cc-31a2b75c6911-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352918 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cd6491b-4078-4652-98cc-31a2b75c6911-cnibin\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.352944 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e864e8ef-600f-4d42-8202-bd16f493e302-ovnkube-config\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353018 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e864e8ef-600f-4d42-8202-bd16f493e302-ovn-node-metrics-cert\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353034 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9ef7c98-636c-443d-bb38-bdd4d0ab3a58-host-slash\") pod \"iptables-alerter-mqng4\" (UID: \"f9ef7c98-636c-443d-bb38-bdd4d0ab3a58\") " pod="openshift-network-operator/iptables-alerter-mqng4" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66d8dd77-971b-4a83-ab58-b49b09d40f7b-konnectivity-ca\") pod \"konnectivity-agent-9vswh\" (UID: \"66d8dd77-971b-4a83-ab58-b49b09d40f7b\") " pod="kube-system/konnectivity-agent-9vswh" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6070593-232f-4a38-b8e2-3380c4abe49e-hosts-file\") pod \"node-resolver-fxvl4\" (UID: \"d6070593-232f-4a38-b8e2-3380c4abe49e\") " pod="openshift-dns/node-resolver-fxvl4" Sep 17 23:06:15.353589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353088 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cd6491b-4078-4652-98cc-31a2b75c6911-system-cni-dir\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-cni-netd\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353149 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjvt\" (UniqueName: \"kubernetes.io/projected/064ea6f5-bd51-42ec-825e-25f7ea4a6723-kube-api-access-crjvt\") pod \"node-ca-l5s9l\" (UID: \"064ea6f5-bd51-42ec-825e-25f7ea4a6723\") " pod="openshift-image-registry/node-ca-l5s9l" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353191 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cd6491b-4078-4652-98cc-31a2b75c6911-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353229 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/064ea6f5-bd51-42ec-825e-25f7ea4a6723-host\") pod \"node-ca-l5s9l\" (UID: \"064ea6f5-bd51-42ec-825e-25f7ea4a6723\") " pod="openshift-image-registry/node-ca-l5s9l" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353343 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/064ea6f5-bd51-42ec-825e-25f7ea4a6723-host\") pod \"node-ca-l5s9l\" (UID: \"064ea6f5-bd51-42ec-825e-25f7ea4a6723\") " pod="openshift-image-registry/node-ca-l5s9l" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353376 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9ef7c98-636c-443d-bb38-bdd4d0ab3a58-iptables-alerter-script\") pod \"iptables-alerter-mqng4\" (UID: \"f9ef7c98-636c-443d-bb38-bdd4d0ab3a58\") " pod="openshift-network-operator/iptables-alerter-mqng4" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353408 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4cd6491b-4078-4652-98cc-31a2b75c6911-cni-binary-copy\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353447 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353461 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9ef7c98-636c-443d-bb38-bdd4d0ab3a58-host-slash\") pod \"iptables-alerter-mqng4\" (UID: \"f9ef7c98-636c-443d-bb38-bdd4d0ab3a58\") " pod="openshift-network-operator/iptables-alerter-mqng4" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4cd6491b-4078-4652-98cc-31a2b75c6911-system-cni-dir\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353591 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e864e8ef-600f-4d42-8202-bd16f493e302-host-cni-netd\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353651 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6070593-232f-4a38-b8e2-3380c4abe49e-hosts-file\") pod \"node-resolver-fxvl4\" (UID: \"d6070593-232f-4a38-b8e2-3380c4abe49e\") " pod="openshift-dns/node-resolver-fxvl4" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353710 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4cd6491b-4078-4652-98cc-31a2b75c6911-cnibin\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.353948 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e864e8ef-600f-4d42-8202-bd16f493e302-env-overrides\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.354017 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4cd6491b-4078-4652-98cc-31a2b75c6911-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.354099 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66d8dd77-971b-4a83-ab58-b49b09d40f7b-konnectivity-ca\") pod \"konnectivity-agent-9vswh\" (UID: \"66d8dd77-971b-4a83-ab58-b49b09d40f7b\") " pod="kube-system/konnectivity-agent-9vswh" Sep 17 23:06:15.354348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.354149 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4cd6491b-4078-4652-98cc-31a2b75c6911-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.354849 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.354321 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/064ea6f5-bd51-42ec-825e-25f7ea4a6723-serviceca\") pod \"node-ca-l5s9l\" (UID: \"064ea6f5-bd51-42ec-825e-25f7ea4a6723\") " pod="openshift-image-registry/node-ca-l5s9l" Sep 17 23:06:15.354849 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.354368 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e864e8ef-600f-4d42-8202-bd16f493e302-ovnkube-script-lib\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.354849 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.354566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66d8dd77-971b-4a83-ab58-b49b09d40f7b-agent-certs\") pod \"konnectivity-agent-9vswh\" (UID: \"66d8dd77-971b-4a83-ab58-b49b09d40f7b\") " pod="kube-system/konnectivity-agent-9vswh" Sep 17 23:06:15.354849 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.354743 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e864e8ef-600f-4d42-8202-bd16f493e302-ovnkube-config\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.355598 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.355578 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e864e8ef-600f-4d42-8202-bd16f493e302-ovn-node-metrics-cert\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.363689 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.363662 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd8k9\" (UniqueName: \"kubernetes.io/projected/d6070593-232f-4a38-b8e2-3380c4abe49e-kube-api-access-gd8k9\") pod \"node-resolver-fxvl4\" (UID: \"d6070593-232f-4a38-b8e2-3380c4abe49e\") " pod="openshift-dns/node-resolver-fxvl4" Sep 17 23:06:15.364229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.364198 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctjwn\" (UniqueName: \"kubernetes.io/projected/e864e8ef-600f-4d42-8202-bd16f493e302-kube-api-access-ctjwn\") pod \"ovnkube-node-ljnfz\" (UID: \"e864e8ef-600f-4d42-8202-bd16f493e302\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.364474 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.364454 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5w65\" (UniqueName: \"kubernetes.io/projected/f9ef7c98-636c-443d-bb38-bdd4d0ab3a58-kube-api-access-q5w65\") pod \"iptables-alerter-mqng4\" (UID: \"f9ef7c98-636c-443d-bb38-bdd4d0ab3a58\") " pod="openshift-network-operator/iptables-alerter-mqng4" Sep 17 23:06:15.366038 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.365662 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crjvt\" (UniqueName: \"kubernetes.io/projected/064ea6f5-bd51-42ec-825e-25f7ea4a6723-kube-api-access-crjvt\") pod \"node-ca-l5s9l\" (UID: \"064ea6f5-bd51-42ec-825e-25f7ea4a6723\") " pod="openshift-image-registry/node-ca-l5s9l" Sep 17 23:06:15.366826 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.366797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcd75\" (UniqueName: \"kubernetes.io/projected/4cd6491b-4078-4652-98cc-31a2b75c6911-kube-api-access-lcd75\") pod \"multus-additional-cni-plugins-lx2wl\" (UID: \"4cd6491b-4078-4652-98cc-31a2b75c6911\") " pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.433615 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:15.433593 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac9bbdb0f7baee4c4f7c0e8cb2517fc.slice/crio-cc33a1767756395759b7bfbfae26a6285eb3318d291249f9e9c0aed27836c7b9 WatchSource:0}: Error finding container cc33a1767756395759b7bfbfae26a6285eb3318d291249f9e9c0aed27836c7b9: Status 404 returned error can't find the container with id cc33a1767756395759b7bfbfae26a6285eb3318d291249f9e9c0aed27836c7b9 Sep 17 23:06:15.438452 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.438439 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 23:06:15.456249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.456226 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" Sep 17 23:06:15.463606 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:15.463546 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373402ca_a06b_4911_8c75_30d18162514a.slice/crio-44f6545bdeb78af56b9fe9dc565bb8ebd58a7b54c8908efe023c2ec3613ed78b WatchSource:0}: Error finding container 44f6545bdeb78af56b9fe9dc565bb8ebd58a7b54c8908efe023c2ec3613ed78b: Status 404 returned error can't find the container with id 44f6545bdeb78af56b9fe9dc565bb8ebd58a7b54c8908efe023c2ec3613ed78b Sep 17 23:06:15.477363 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.477345 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-swdt6" Sep 17 23:06:15.483159 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.483138 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nvk6c" Sep 17 23:06:15.483641 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:15.483620 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91666ad_1380_411e_8615_bdf527db6ef3.slice/crio-2d8866d12c32063363819af8daeaba4bd08137f4a164886cf8aaf9c54b300536 WatchSource:0}: Error finding container 2d8866d12c32063363819af8daeaba4bd08137f4a164886cf8aaf9c54b300536: Status 404 returned error can't find the container with id 2d8866d12c32063363819af8daeaba4bd08137f4a164886cf8aaf9c54b300536 Sep 17 23:06:15.490340 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:15.490321 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ae3825_31b7_4c22_bf96_d6ee2c98f2c9.slice/crio-2d20a345f08395da42a9524d1ff2dde71b03527cac1b879f240ccc4cdd651790 WatchSource:0}: Error finding container 2d20a345f08395da42a9524d1ff2dde71b03527cac1b879f240ccc4cdd651790: Status 404 returned error can't find the container with id 2d20a345f08395da42a9524d1ff2dde71b03527cac1b879f240ccc4cdd651790 Sep 17 23:06:15.508756 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.508735 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mqng4" Sep 17 23:06:15.515231 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:15.515214 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ef7c98_636c_443d_bb38_bdd4d0ab3a58.slice/crio-08d7ed0056d9d73e224fc6d0d780dd5e1a8dbbbd72cffdde7e95817b006ebe59 WatchSource:0}: Error finding container 08d7ed0056d9d73e224fc6d0d780dd5e1a8dbbbd72cffdde7e95817b006ebe59: Status 404 returned error can't find the container with id 08d7ed0056d9d73e224fc6d0d780dd5e1a8dbbbd72cffdde7e95817b006ebe59 Sep 17 23:06:15.526715 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.526699 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:15.532622 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:15.532604 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode864e8ef_600f_4d42_8202_bd16f493e302.slice/crio-ea26b0bf0f9604fe6c21dfb02f0e07616ae94d3aac6dd50155acbc3ea853044f WatchSource:0}: Error finding container ea26b0bf0f9604fe6c21dfb02f0e07616ae94d3aac6dd50155acbc3ea853044f: Status 404 returned error can't find the container with id ea26b0bf0f9604fe6c21dfb02f0e07616ae94d3aac6dd50155acbc3ea853044f Sep 17 23:06:15.543750 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.543732 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9vswh" Sep 17 23:06:15.551329 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:15.551311 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d8dd77_971b_4a83_ab58_b49b09d40f7b.slice/crio-96c914c8678e1397afbcff2841c5f972a6b1d697ab33588c525818297ab51854 WatchSource:0}: Error finding container 96c914c8678e1397afbcff2841c5f972a6b1d697ab33588c525818297ab51854: Status 404 returned error can't find the container with id 96c914c8678e1397afbcff2841c5f972a6b1d697ab33588c525818297ab51854 Sep 17 23:06:15.567631 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.567612 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fxvl4" Sep 17 23:06:15.573872 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:15.573856 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6070593_232f_4a38_b8e2_3380c4abe49e.slice/crio-af34d1fe72dde43cac59395f78627166f72702c55b4c5047f24c34780a1a6036 WatchSource:0}: Error finding container af34d1fe72dde43cac59395f78627166f72702c55b4c5047f24c34780a1a6036: Status 404 returned error can't find the container with id af34d1fe72dde43cac59395f78627166f72702c55b4c5047f24c34780a1a6036 Sep 17 23:06:15.585421 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.585406 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l5s9l" Sep 17 23:06:15.590047 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.589983 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lx2wl" Sep 17 23:06:15.592035 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:15.592016 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064ea6f5_bd51_42ec_825e_25f7ea4a6723.slice/crio-14f89636cb53b04fe071fe802ba39719b9744b658bba90d03f33bec5c9856608 WatchSource:0}: Error finding container 14f89636cb53b04fe071fe802ba39719b9744b658bba90d03f33bec5c9856608: Status 404 returned error can't find the container with id 14f89636cb53b04fe071fe802ba39719b9744b658bba90d03f33bec5c9856608 Sep 17 23:06:15.597007 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:15.596985 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd6491b_4078_4652_98cc_31a2b75c6911.slice/crio-b6f1bc69b3a15d97143f20cca8a23a60992b5861bf6e4ee12e6b69b6ed9258f5 WatchSource:0}: Error finding container b6f1bc69b3a15d97143f20cca8a23a60992b5861bf6e4ee12e6b69b6ed9258f5: Status 404 returned error can't find the container with id b6f1bc69b3a15d97143f20cca8a23a60992b5861bf6e4ee12e6b69b6ed9258f5 Sep 17 23:06:15.663848 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.663827 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:15.720284 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:15.720260 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ebc53fbb6af3fc3b4be0312595c4c9.slice/crio-b4198a4fbefde1fd78fa89864a2c025d5c8c402c910c9abdc89d62d427eb64a3 WatchSource:0}: Error finding container b4198a4fbefde1fd78fa89864a2c025d5c8c402c910c9abdc89d62d427eb64a3: Status 404 returned error can't find the container with id b4198a4fbefde1fd78fa89864a2c025d5c8c402c910c9abdc89d62d427eb64a3 Sep 17 23:06:15.757279 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.757245 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlff5\" (UniqueName: \"kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5\") pod \"network-check-target-sq8mp\" (UID: \"13108b6b-4ed6-4ca0-893a-fd49f1ae5b78\") " pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:15.757385 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.757293 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:15.757385 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:15.757316 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:15.757472 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.757400 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:15.757472 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.757403 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:15.757472 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.757426 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:15.757472 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.757436 2559 projected.go:196] Error preparing data for projected volume kube-api-access-xlff5 for pod openshift-network-diagnostics/network-check-target-sq8mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:15.757472 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.757444 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs podName:ea1ba703-ed87-422b-a256-bc96859795af nodeName:}" failed. No retries permitted until 2026-09-17 23:06:16.757427807 +0000 UTC m=+4.874907236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs") pod "network-metrics-daemon-vk29c" (UID: "ea1ba703-ed87-422b-a256-bc96859795af") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:15.757633 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.757401 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:15.757633 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.757477 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5 podName:13108b6b-4ed6-4ca0-893a-fd49f1ae5b78 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:16.757462764 +0000 UTC m=+4.874942209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlff5" (UniqueName: "kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5") pod "network-check-target-sq8mp" (UID: "13108b6b-4ed6-4ca0-893a-fd49f1ae5b78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:15.757633 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:15.757538 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret podName:4512e542-b8f7-4833-b2a5-9812a90f43c6 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:16.757514327 +0000 UTC m=+4.874993756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret") pod "global-pull-secret-syncer-92l5f" (UID: "4512e542-b8f7-4833-b2a5-9812a90f43c6") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:16.214861 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.214756 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:01:15 +0000 UTC" deadline="2028-04-16 11:39:44.222668349 +0000 UTC" Sep 17 23:06:16.214861 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.214816 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13836h33m28.007856574s" Sep 17 23:06:16.283696 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.283664 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:16.283862 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:16.283799 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:16.297673 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.297623 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx2wl" event={"ID":"4cd6491b-4078-4652-98cc-31a2b75c6911","Type":"ContainerStarted","Data":"b6f1bc69b3a15d97143f20cca8a23a60992b5861bf6e4ee12e6b69b6ed9258f5"} Sep 17 23:06:16.319815 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.319778 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mqng4" event={"ID":"f9ef7c98-636c-443d-bb38-bdd4d0ab3a58","Type":"ContainerStarted","Data":"08d7ed0056d9d73e224fc6d0d780dd5e1a8dbbbd72cffdde7e95817b006ebe59"} Sep 17 23:06:16.339530 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.339497 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" event={"ID":"373402ca-a06b-4911-8c75-30d18162514a","Type":"ContainerStarted","Data":"44f6545bdeb78af56b9fe9dc565bb8ebd58a7b54c8908efe023c2ec3613ed78b"} Sep 17 23:06:16.345071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.345041 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" event={"ID":"9ac9bbdb0f7baee4c4f7c0e8cb2517fc","Type":"ContainerStarted","Data":"cc33a1767756395759b7bfbfae26a6285eb3318d291249f9e9c0aed27836c7b9"} Sep 17 23:06:16.348264 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.348217 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l5s9l" event={"ID":"064ea6f5-bd51-42ec-825e-25f7ea4a6723","Type":"ContainerStarted","Data":"14f89636cb53b04fe071fe802ba39719b9744b658bba90d03f33bec5c9856608"} Sep 17 23:06:16.352854 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.352831 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fxvl4" event={"ID":"d6070593-232f-4a38-b8e2-3380c4abe49e","Type":"ContainerStarted","Data":"af34d1fe72dde43cac59395f78627166f72702c55b4c5047f24c34780a1a6036"} Sep 17 23:06:16.368930 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.368855 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9vswh" event={"ID":"66d8dd77-971b-4a83-ab58-b49b09d40f7b","Type":"ContainerStarted","Data":"96c914c8678e1397afbcff2841c5f972a6b1d697ab33588c525818297ab51854"} Sep 17 23:06:16.371319 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.371298 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:16.377055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.377008 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" event={"ID":"e864e8ef-600f-4d42-8202-bd16f493e302","Type":"ContainerStarted","Data":"ea26b0bf0f9604fe6c21dfb02f0e07616ae94d3aac6dd50155acbc3ea853044f"} Sep 17 23:06:16.392086 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.391974 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nvk6c" event={"ID":"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9","Type":"ContainerStarted","Data":"2d20a345f08395da42a9524d1ff2dde71b03527cac1b879f240ccc4cdd651790"} Sep 17 23:06:16.403745 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.403604 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-swdt6" event={"ID":"e91666ad-1380-411e-8615-bdf527db6ef3","Type":"ContainerStarted","Data":"2d8866d12c32063363819af8daeaba4bd08137f4a164886cf8aaf9c54b300536"} Sep 17 23:06:16.425491 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.425449 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" event={"ID":"a4ebc53fbb6af3fc3b4be0312595c4c9","Type":"ContainerStarted","Data":"b4198a4fbefde1fd78fa89864a2c025d5c8c402c910c9abdc89d62d427eb64a3"} Sep 17 23:06:16.780752 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.780717 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlff5\" (UniqueName: \"kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5\") pod \"network-check-target-sq8mp\" (UID: \"13108b6b-4ed6-4ca0-893a-fd49f1ae5b78\") " pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:16.780942 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.780778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:16.780942 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:16.780809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:16.781073 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:16.780947 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:16.781073 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:16.781009 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs podName:ea1ba703-ed87-422b-a256-bc96859795af nodeName:}" failed. No retries permitted until 2026-09-17 23:06:18.780990036 +0000 UTC m=+6.898469467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs") pod "network-metrics-daemon-vk29c" (UID: "ea1ba703-ed87-422b-a256-bc96859795af") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:16.781650 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:16.781422 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:16.781650 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:16.781445 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:16.781650 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:16.781457 2559 projected.go:196] Error preparing data for projected volume kube-api-access-xlff5 for pod openshift-network-diagnostics/network-check-target-sq8mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:16.781650 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:16.781504 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5 podName:13108b6b-4ed6-4ca0-893a-fd49f1ae5b78 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:18.781488312 +0000 UTC m=+6.898967744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlff5" (UniqueName: "kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5") pod "network-check-target-sq8mp" (UID: "13108b6b-4ed6-4ca0-893a-fd49f1ae5b78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:16.781650 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:16.781568 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:16.781650 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:16.781599 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret podName:4512e542-b8f7-4833-b2a5-9812a90f43c6 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:18.781588772 +0000 UTC m=+6.899068208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret") pod "global-pull-secret-syncer-92l5f" (UID: "4512e542-b8f7-4833-b2a5-9812a90f43c6") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:17.215987 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:17.215884 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:01:15 +0000 UTC" deadline="2028-07-04 20:36:44.847827412 +0000 UTC" Sep 17 23:06:17.215987 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:17.215938 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15741h30m27.631894709s" Sep 17 23:06:17.280975 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:17.280924 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:17.281205 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:17.281065 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:17.281205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:17.281153 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:17.281321 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:17.281253 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:17.859068 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:17.859037 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:17.946130 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:17.946091 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:18.285763 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:18.285684 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:18.286224 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:18.285811 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:18.802408 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:18.802242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlff5\" (UniqueName: \"kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5\") pod \"network-check-target-sq8mp\" (UID: \"13108b6b-4ed6-4ca0-893a-fd49f1ae5b78\") " pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:18.802408 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:18.802315 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:18.802408 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:18.802348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:18.802695 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:18.802480 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:18.802695 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:18.802546 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs podName:ea1ba703-ed87-422b-a256-bc96859795af nodeName:}" failed. No retries permitted until 2026-09-17 23:06:22.802525951 +0000 UTC m=+10.920005400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs") pod "network-metrics-daemon-vk29c" (UID: "ea1ba703-ed87-422b-a256-bc96859795af") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:18.803007 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:18.802985 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:18.803093 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:18.803012 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:18.803093 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:18.803024 2559 projected.go:196] Error preparing data for projected volume kube-api-access-xlff5 for pod openshift-network-diagnostics/network-check-target-sq8mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:18.803093 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:18.803072 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5 podName:13108b6b-4ed6-4ca0-893a-fd49f1ae5b78 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:22.803055506 +0000 UTC m=+10.920534938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlff5" (UniqueName: "kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5") pod "network-check-target-sq8mp" (UID: "13108b6b-4ed6-4ca0-893a-fd49f1ae5b78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:18.803260 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:18.803135 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:18.803260 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:18.803172 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret podName:4512e542-b8f7-4833-b2a5-9812a90f43c6 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:22.803160706 +0000 UTC m=+10.920640136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret") pod "global-pull-secret-syncer-92l5f" (UID: "4512e542-b8f7-4833-b2a5-9812a90f43c6") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:19.281856 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:19.281358 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:19.281856 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:19.281505 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:19.281856 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:19.281698 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:19.281856 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:19.281815 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:20.281784 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:20.281180 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:20.281784 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:20.281360 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:21.280675 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:21.280564 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:21.280878 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:21.280748 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:21.281161 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:21.280576 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:21.281161 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:21.281118 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:22.280758 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:22.280717 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:22.281269 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:22.280912 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:22.838213 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:22.838177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlff5\" (UniqueName: \"kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5\") pod \"network-check-target-sq8mp\" (UID: \"13108b6b-4ed6-4ca0-893a-fd49f1ae5b78\") " pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:22.838427 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:22.838244 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:22.838427 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:22.838280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:22.838427 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:22.838406 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:22.838615 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:22.838463 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:22.838615 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:22.838489 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:22.838615 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:22.838502 2559 projected.go:196] Error preparing data for projected volume kube-api-access-xlff5 for pod openshift-network-diagnostics/network-check-target-sq8mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:22.838615 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:22.838474 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs podName:ea1ba703-ed87-422b-a256-bc96859795af nodeName:}" failed. No retries permitted until 2026-09-17 23:06:30.838454675 +0000 UTC m=+18.955934110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs") pod "network-metrics-daemon-vk29c" (UID: "ea1ba703-ed87-422b-a256-bc96859795af") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:22.838615 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:22.838564 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5 podName:13108b6b-4ed6-4ca0-893a-fd49f1ae5b78 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:30.838546893 +0000 UTC m=+18.956026339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlff5" (UniqueName: "kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5") pod "network-check-target-sq8mp" (UID: "13108b6b-4ed6-4ca0-893a-fd49f1ae5b78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:22.838615 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:22.838406 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:22.838615 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:22.838602 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret podName:4512e542-b8f7-4833-b2a5-9812a90f43c6 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:30.838592802 +0000 UTC m=+18.956072245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret") pod "global-pull-secret-syncer-92l5f" (UID: "4512e542-b8f7-4833-b2a5-9812a90f43c6") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:23.281211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:23.281172 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:23.281211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:23.281207 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:23.281743 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:23.281323 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:23.281743 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:23.281462 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:24.281706 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:24.281675 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:24.282200 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:24.281811 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:25.281163 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:25.281126 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:25.281163 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:25.281148 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:25.281417 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:25.281261 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:25.281417 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:25.281391 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:26.280754 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:26.280718 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:26.281305 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:26.280858 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:27.281196 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:27.281150 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:27.281196 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:27.281193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:27.281757 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:27.281295 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:27.281757 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:27.281413 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:28.281074 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:28.281036 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:28.281340 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:28.281186 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:29.280652 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:29.280617 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:29.280824 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:29.280626 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:29.280824 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:29.280724 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:29.280824 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:29.280808 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:30.281361 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:30.281338 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:30.281668 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:30.281444 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:30.901273 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:30.901234 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlff5\" (UniqueName: \"kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5\") pod \"network-check-target-sq8mp\" (UID: \"13108b6b-4ed6-4ca0-893a-fd49f1ae5b78\") " pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:30.901273 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:30.901289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:30.901489 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:30.901320 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:30.901489 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:30.901461 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:30.901489 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:30.901483 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:30.901604 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:30.901498 2559 projected.go:196] Error preparing data for projected volume kube-api-access-xlff5 for pod openshift-network-diagnostics/network-check-target-sq8mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:30.901604 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:30.901526 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:30.901604 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:30.901564 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5 podName:13108b6b-4ed6-4ca0-893a-fd49f1ae5b78 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:46.901545769 +0000 UTC m=+35.019025199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlff5" (UniqueName: "kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5") pod "network-check-target-sq8mp" (UID: "13108b6b-4ed6-4ca0-893a-fd49f1ae5b78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:30.901604 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:30.901594 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret podName:4512e542-b8f7-4833-b2a5-9812a90f43c6 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:46.901575548 +0000 UTC m=+35.019054991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret") pod "global-pull-secret-syncer-92l5f" (UID: "4512e542-b8f7-4833-b2a5-9812a90f43c6") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:30.901604 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:30.901534 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:30.901796 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:30.901637 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs podName:ea1ba703-ed87-422b-a256-bc96859795af nodeName:}" failed. No retries permitted until 2026-09-17 23:06:46.901626788 +0000 UTC m=+35.019106219 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs") pod "network-metrics-daemon-vk29c" (UID: "ea1ba703-ed87-422b-a256-bc96859795af") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:31.280765 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:31.280724 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:31.281032 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:31.280848 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:31.281115 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:31.280732 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:31.281251 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:31.281229 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:32.280678 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:32.280638 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:32.281130 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:32.280770 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:33.280502 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:33.280459 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:33.280690 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:33.280599 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:33.280690 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:33.280656 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:33.281081 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:33.280796 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:34.283791 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:34.282534 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:34.283791 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:34.283337 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:34.479002 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:34.478774 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" event={"ID":"e864e8ef-600f-4d42-8202-bd16f493e302","Type":"ContainerStarted","Data":"a2ce99fcff9381b045ffa1bf14890731982e6c945a143abe4751c531c5389bd0"} Sep 17 23:06:34.479117 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:34.479058 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" event={"ID":"e864e8ef-600f-4d42-8202-bd16f493e302","Type":"ContainerStarted","Data":"30e6619909c545b61b41f2aa6e803e6696c9b14222bb9cc690bcf792664f50a5"} Sep 17 23:06:34.480248 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:34.480197 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nvk6c" event={"ID":"99ae3825-31b7-4c22-bf96-d6ee2c98f2c9","Type":"ContainerStarted","Data":"13558e2ad5118edb3027fde9ee8f19de62abd740f74d61a15f9bedd14a59c08e"} Sep 17 23:06:34.481621 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:34.481585 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-swdt6" event={"ID":"e91666ad-1380-411e-8615-bdf527db6ef3","Type":"ContainerStarted","Data":"a147d29549e2b0aa2a501621075132847f6596bf67f1ed4729a271a4726a3a3c"} Sep 17 23:06:34.482813 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:34.482792 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" event={"ID":"9ac9bbdb0f7baee4c4f7c0e8cb2517fc","Type":"ContainerStarted","Data":"fade2bfd1d32bf719718f8f55fa0762c74d5548ae25fdf71c2a86127a81ce7bb"} Sep 17 23:06:34.500838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:34.500790 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nvk6c" podStartSLOduration=1.7241697 podStartE2EDuration="20.500774684s" totalImagesPullingTime="18.776604984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:34.500120465 +0000 UTC m=+22.617599941" watchObservedRunningTime="2026-09-17 23:06:34.500774684 +0000 UTC m=+22.618254134" Sep 17 23:06:34.516546 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:34.516496 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-swdt6" podStartSLOduration=2.003397074 podStartE2EDuration="20.516471829s" totalImagesPullingTime="18.513074755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:34.516334105 +0000 UTC m=+22.633813578" watchObservedRunningTime="2026-09-17 23:06:34.516471829 +0000 UTC m=+22.633951280" Sep 17 23:06:34.532012 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:34.531970 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" podStartSLOduration=19.531953525 podStartE2EDuration="19.531953525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:34.531489225 +0000 UTC m=+22.648968689" watchObservedRunningTime="2026-09-17 23:06:34.531953525 +0000 UTC m=+22.649432976" Sep 17 23:06:35.280848 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.280810 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:35.281021 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:35.280980 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:35.281082 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.281037 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:35.281190 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:35.281170 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:35.459853 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.459688 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 23:06:35.487266 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.487230 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" event={"ID":"373402ca-a06b-4911-8c75-30d18162514a","Type":"ContainerStarted","Data":"86de7addddea4c86e4ed586dd1c9b3b24be1ebff174b2bde2341f29db6bdecc7"} Sep 17 23:06:35.487405 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.487292 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" event={"ID":"373402ca-a06b-4911-8c75-30d18162514a","Type":"ContainerStarted","Data":"1b27bf8623d825c4e78f8874553ff993caefc66b33323634d9359c31dd3f0b4f"} Sep 17 23:06:35.488666 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.488635 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l5s9l" event={"ID":"064ea6f5-bd51-42ec-825e-25f7ea4a6723","Type":"ContainerStarted","Data":"fac022e1d4ca0d1a12183839845cfb961837c884a74d2c3b6686835e2fb07d90"} Sep 17 23:06:35.490055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.490024 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fxvl4" event={"ID":"d6070593-232f-4a38-b8e2-3380c4abe49e","Type":"ContainerStarted","Data":"95e6b5ceda00aaf413716c0c0335f80ce3faaec1fe27e24e19f39060e19bc38f"} Sep 17 23:06:35.491446 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.491426 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9vswh" event={"ID":"66d8dd77-971b-4a83-ab58-b49b09d40f7b","Type":"ContainerStarted","Data":"a8fec314ff72dbe8234d1d049b435c79e397fa9a409540d747e6ad84469b6048"} Sep 17 23:06:35.494392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.494369 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" event={"ID":"e864e8ef-600f-4d42-8202-bd16f493e302","Type":"ContainerStarted","Data":"e7004ea7cffa3bb5c45b9d45afa960ee2404b1e58cba0bfe293e86ac23ca12e2"} Sep 17 23:06:35.494505 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.494431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" event={"ID":"e864e8ef-600f-4d42-8202-bd16f493e302","Type":"ContainerStarted","Data":"8c1f9229aef47bb625257cedf14855b1190f5f530cec5368127c592702b430b9"} Sep 17 23:06:35.494505 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.494463 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" event={"ID":"e864e8ef-600f-4d42-8202-bd16f493e302","Type":"ContainerStarted","Data":"5aaaa9ee132bbb428819eb321c1ce60c8dddc96cc8a70ebee3c13953395ccc6c"} Sep 17 23:06:35.494505 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.494498 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" event={"ID":"e864e8ef-600f-4d42-8202-bd16f493e302","Type":"ContainerStarted","Data":"5aa942b9345b518bfc09898eff14ab3e1de9ac59ff64c92164d50552d35b7841"} Sep 17 23:06:35.495803 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.495782 2559 generic.go:356] "Generic (PLEG): container finished" podID="a4ebc53fbb6af3fc3b4be0312595c4c9" containerID="5133a53215d286f8615436eccdcb26f4df0c597d8eeb49c5e32e99725930ec42" exitCode=0 Sep 17 23:06:35.495895 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.495846 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" event={"ID":"a4ebc53fbb6af3fc3b4be0312595c4c9","Type":"ContainerDied","Data":"5133a53215d286f8615436eccdcb26f4df0c597d8eeb49c5e32e99725930ec42"} Sep 17 23:06:35.497562 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.497540 2559 generic.go:356] "Generic (PLEG): container finished" podID="4cd6491b-4078-4652-98cc-31a2b75c6911" containerID="fedfc381cbe8096b8ba822859e8c8cb119498ac2f0f9ec6c567eb9175e2ae87c" exitCode=0 Sep 17 23:06:35.497649 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.497613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx2wl" event={"ID":"4cd6491b-4078-4652-98cc-31a2b75c6911","Type":"ContainerDied","Data":"fedfc381cbe8096b8ba822859e8c8cb119498ac2f0f9ec6c567eb9175e2ae87c"} Sep 17 23:06:35.499222 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.499198 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mqng4" event={"ID":"f9ef7c98-636c-443d-bb38-bdd4d0ab3a58","Type":"ContainerStarted","Data":"b24c5a534783794abc01417adbbd20ad135d4ec190f9aa2ee6c8cc7951693819"} Sep 17 23:06:35.503447 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.503406 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l5s9l" podStartSLOduration=3.099899313 podStartE2EDuration="21.503391592s" totalImagesPullingTime="18.403492279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:35.503088348 +0000 UTC m=+23.620567815" watchObservedRunningTime="2026-09-17 23:06:35.503391592 +0000 UTC m=+23.620871042" Sep 17 23:06:35.534958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.534891 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9vswh" podStartSLOduration=6.802408261 podStartE2EDuration="21.534874901s" totalImagesPullingTime="14.73246664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:35.534574151 +0000 UTC m=+23.652053624" watchObservedRunningTime="2026-09-17 23:06:35.534874901 +0000 UTC m=+23.652354352" Sep 17 23:06:35.551215 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.551167 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fxvl4" podStartSLOduration=3.129811503 podStartE2EDuration="21.551150853s" totalImagesPullingTime="18.42133935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:35.550975568 +0000 UTC m=+23.668455032" watchObservedRunningTime="2026-09-17 23:06:35.551150853 +0000 UTC m=+23.668630303" Sep 17 23:06:35.566084 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:35.566043 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mqng4" podStartSLOduration=3.0854854720000002 podStartE2EDuration="21.56602599s" totalImagesPullingTime="18.480540518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:35.565541704 +0000 UTC m=+23.683021178" watchObservedRunningTime="2026-09-17 23:06:35.56602599 +0000 UTC m=+23.683505441" Sep 17 23:06:36.222170 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:36.222054 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:06:35.459848553Z","UUID":"f71099dc-50c3-468e-a38e-758447626fab","Handler":null,"Name":"","Endpoint":""} Sep 17 23:06:36.224088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:36.224062 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 23:06:36.224211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:36.224103 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 23:06:36.284952 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:36.284924 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:36.285117 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:36.285029 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:36.504304 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:36.504211 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" event={"ID":"a4ebc53fbb6af3fc3b4be0312595c4c9","Type":"ContainerStarted","Data":"2096a2d69487293850a243e4356394590ddf90be88fc9495cff4a251ee51db6f"} Sep 17 23:06:36.519141 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:36.519091 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" podStartSLOduration=21.519074486 podStartE2EDuration="21.519074486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:36.518556867 +0000 UTC m=+24.636036318" watchObservedRunningTime="2026-09-17 23:06:36.519074486 +0000 UTC m=+24.636553939" Sep 17 23:06:37.281480 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:37.281446 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:37.281708 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:37.281573 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:37.281708 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:37.281627 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:37.281849 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:37.281733 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:37.508849 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:37.508810 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" event={"ID":"373402ca-a06b-4911-8c75-30d18162514a","Type":"ContainerStarted","Data":"2ea72f0932629791fcc53fc4c0d805ed771b69cecfecf248d67b1a159b6a1689"} Sep 17 23:06:37.513066 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:37.513029 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" event={"ID":"e864e8ef-600f-4d42-8202-bd16f493e302","Type":"ContainerStarted","Data":"4ef3ce95cbe7e3b45b4e2b2f1f75cbbc4df76a9a42f2593953b655e836c3cc32"} Sep 17 23:06:37.526636 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:37.526554 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4q5m" podStartSLOduration=3.0924285400000002 podStartE2EDuration="23.526531956s" totalImagesPullingTime="20.434103416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:37.526154104 +0000 UTC m=+25.643633586" watchObservedRunningTime="2026-09-17 23:06:37.526531956 +0000 UTC m=+25.644011411" Sep 17 23:06:38.281285 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:38.281248 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:38.281483 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:38.281380 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:39.281175 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:39.280944 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:39.281678 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:39.280995 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:39.281678 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:39.281226 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:39.281678 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:39.281356 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:39.522572 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:39.522521 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" event={"ID":"e864e8ef-600f-4d42-8202-bd16f493e302","Type":"ContainerStarted","Data":"5fa2e488fb51b65011badaa7d626af0133c4453ee3817beb72b04a4db0c32098"} Sep 17 23:06:39.522820 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:39.522792 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:39.538383 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:39.538311 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:39.553894 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:39.553848 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" podStartSLOduration=7.048174516 podStartE2EDuration="25.553829495s" totalImagesPullingTime="18.505654979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:39.552597001 +0000 UTC m=+27.670076473" watchObservedRunningTime="2026-09-17 23:06:39.553829495 +0000 UTC m=+27.671308946" Sep 17 23:06:39.815529 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:39.815439 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9vswh" Sep 17 23:06:39.816157 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:39.816136 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9vswh" Sep 17 23:06:40.280655 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:40.280616 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:40.280845 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:40.280745 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:40.525371 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:40.525314 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:40.525833 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:40.525413 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:40.525833 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:40.525458 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9vswh" Sep 17 23:06:40.525977 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:40.525947 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9vswh" Sep 17 23:06:40.542880 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:40.542805 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:06:40.947778 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:40.947738 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-92l5f"] Sep 17 23:06:40.947975 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:40.947914 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:40.948051 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:40.948025 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:40.950503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:40.950474 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vk29c"] Sep 17 23:06:40.950626 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:40.950599 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:40.950743 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:40.950708 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:40.953376 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:40.953353 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sq8mp"] Sep 17 23:06:40.953488 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:40.953459 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:40.953580 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:40.953551 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:42.281473 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:42.281444 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:42.282082 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:42.281444 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:42.282082 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:42.281570 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:42.282082 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:42.281616 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:42.531466 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:42.531432 2559 generic.go:356] "Generic (PLEG): container finished" podID="4cd6491b-4078-4652-98cc-31a2b75c6911" containerID="c8028398047d447004c591890bbae45e42bc2ef399568391e49614f2193adb3c" exitCode=0 Sep 17 23:06:42.531616 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:42.531519 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx2wl" event={"ID":"4cd6491b-4078-4652-98cc-31a2b75c6911","Type":"ContainerDied","Data":"c8028398047d447004c591890bbae45e42bc2ef399568391e49614f2193adb3c"} Sep 17 23:06:43.281332 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:43.281281 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:43.281513 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:43.281420 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:44.281652 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:44.281446 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:44.282183 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:44.281516 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:44.282183 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:44.281717 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:44.282183 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:44.281794 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:44.538073 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:44.537987 2559 generic.go:356] "Generic (PLEG): container finished" podID="4cd6491b-4078-4652-98cc-31a2b75c6911" containerID="945a14a880e92d5e96c84540384de91b27c96e8ea6e07032d8bbbacd74bda905" exitCode=0 Sep 17 23:06:44.538073 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:44.538046 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx2wl" event={"ID":"4cd6491b-4078-4652-98cc-31a2b75c6911","Type":"ContainerDied","Data":"945a14a880e92d5e96c84540384de91b27c96e8ea6e07032d8bbbacd74bda905"} Sep 17 23:06:45.281371 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:45.281332 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:45.281553 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:45.281479 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq8mp" podUID="13108b6b-4ed6-4ca0-893a-fd49f1ae5b78" Sep 17 23:06:45.542712 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:45.542624 2559 generic.go:356] "Generic (PLEG): container finished" podID="4cd6491b-4078-4652-98cc-31a2b75c6911" containerID="0e9f6b3a03ff1c300acb742c510433c3b6e8d7f1fe48cf6f106d0236b1f4e407" exitCode=0 Sep 17 23:06:45.542712 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:45.542663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx2wl" event={"ID":"4cd6491b-4078-4652-98cc-31a2b75c6911","Type":"ContainerDied","Data":"0e9f6b3a03ff1c300acb742c510433c3b6e8d7f1fe48cf6f106d0236b1f4e407"} Sep 17 23:06:46.281528 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:46.281497 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:46.281722 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:46.281615 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vk29c" podUID="ea1ba703-ed87-422b-a256-bc96859795af" Sep 17 23:06:46.281722 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:46.281674 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:46.281845 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:46.281798 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l5f" podUID="4512e542-b8f7-4833-b2a5-9812a90f43c6" Sep 17 23:06:46.923518 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:46.923475 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlff5\" (UniqueName: \"kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5\") pod \"network-check-target-sq8mp\" (UID: \"13108b6b-4ed6-4ca0-893a-fd49f1ae5b78\") " pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:46.923931 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:46.923531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:46.923931 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:46.923550 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:46.923931 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:46.923638 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:46.923931 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:46.923656 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:46.923931 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:46.923685 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:46.923931 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:46.923691 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs podName:ea1ba703-ed87-422b-a256-bc96859795af nodeName:}" failed. No retries permitted until 2026-09-17 23:07:18.923677479 +0000 UTC m=+67.041156907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs") pod "network-metrics-daemon-vk29c" (UID: "ea1ba703-ed87-422b-a256-bc96859795af") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:46.923931 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:46.923695 2559 projected.go:196] Error preparing data for projected volume kube-api-access-xlff5 for pod openshift-network-diagnostics/network-check-target-sq8mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:46.923931 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:46.923741 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5 podName:13108b6b-4ed6-4ca0-893a-fd49f1ae5b78 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:18.923725169 +0000 UTC m=+67.041204610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlff5" (UniqueName: "kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5") pod "network-check-target-sq8mp" (UID: "13108b6b-4ed6-4ca0-893a-fd49f1ae5b78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:46.923931 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:46.923668 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:46.923931 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:46.923823 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret podName:4512e542-b8f7-4833-b2a5-9812a90f43c6 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:18.923801601 +0000 UTC m=+67.041281046 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret") pod "global-pull-secret-syncer-92l5f" (UID: "4512e542-b8f7-4833-b2a5-9812a90f43c6") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:47.144530 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.144476 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeReady" Sep 17 23:06:47.144718 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.144655 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 23:06:47.176069 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.176029 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f64f764df-gtws9"] Sep 17 23:06:47.177996 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.177971 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.180685 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.180661 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 23:06:47.180685 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.180642 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 23:06:47.181006 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.180831 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 23:06:47.181006 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.180831 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cm7l8\"" Sep 17 23:06:47.186570 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.186541 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 23:06:47.191082 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.191060 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f64f764df-gtws9"] Sep 17 23:06:47.194686 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.194658 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2fdkb"] Sep 17 23:06:47.196556 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.196536 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6xlrg"] Sep 17 23:06:47.196671 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.196660 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.198640 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.198619 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6xlrg" Sep 17 23:06:47.199539 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.199523 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vvv69\"" Sep 17 23:06:47.199637 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.199562 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 23:06:47.199966 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.199951 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 23:06:47.201239 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.201220 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 23:06:47.201321 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.201220 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 23:06:47.201321 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.201247 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 23:06:47.201686 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.201671 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-672mp\"" Sep 17 23:06:47.207469 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.207446 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6xlrg"] Sep 17 23:06:47.211428 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.211405 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2fdkb"] Sep 17 23:06:47.280457 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.280427 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:06:47.283046 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.283022 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 23:06:47.283201 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.283023 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kxqn4\"" Sep 17 23:06:47.283442 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.283421 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 23:06:47.325977 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.325865 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjlqp\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-kube-api-access-sjlqp\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.325977 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.325924 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-certificates\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.325977 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.325945 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d16d7679-5919-42cb-b4c7-379fadd35925-cert\") pod \"ingress-canary-6xlrg\" (UID: \"d16d7679-5919-42cb-b4c7-379fadd35925\") " pod="openshift-ingress-canary/ingress-canary-6xlrg" Sep 17 23:06:47.325977 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.325961 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-config-volume\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.326252 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.326026 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnbvt\" (UniqueName: \"kubernetes.io/projected/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-kube-api-access-qnbvt\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.326252 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.326081 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.326252 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.326112 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce53708a-abfa-4b74-b949-2456ce5cd297-image-registry-private-configuration\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.326252 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.326133 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.326252 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.326155 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-tmp-dir\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.326252 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.326204 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce53708a-abfa-4b74-b949-2456ce5cd297-ca-trust-extracted\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.326252 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.326222 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce53708a-abfa-4b74-b949-2456ce5cd297-trusted-ca\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.326441 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.326255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-bound-sa-token\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.326441 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.326270 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrg9q\" (UniqueName: \"kubernetes.io/projected/d16d7679-5919-42cb-b4c7-379fadd35925-kube-api-access-wrg9q\") pod \"ingress-canary-6xlrg\" (UID: \"d16d7679-5919-42cb-b4c7-379fadd35925\") " pod="openshift-ingress-canary/ingress-canary-6xlrg" Sep 17 23:06:47.326441 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.326290 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce53708a-abfa-4b74-b949-2456ce5cd297-installation-pull-secrets\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.427105 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427071 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.427105 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427108 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce53708a-abfa-4b74-b949-2456ce5cd297-image-registry-private-configuration\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.427343 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.427343 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:47.427218 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 23:06:47.427343 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:47.427232 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f64f764df-gtws9: secret "image-registry-tls" not found Sep 17 23:06:47.427343 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:47.427269 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:06:47.427343 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:47.427289 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls podName:ce53708a-abfa-4b74-b949-2456ce5cd297 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:47.927270045 +0000 UTC m=+36.044749524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls") pod "image-registry-7f64f764df-gtws9" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297") : secret "image-registry-tls" not found Sep 17 23:06:47.427343 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427316 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-tmp-dir\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.427343 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:47.427329 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls podName:5b7ac966-8594-4a9f-aa3f-85f12087b2a3 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:47.927312507 +0000 UTC m=+36.044791950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls") pod "dns-default-2fdkb" (UID: "5b7ac966-8594-4a9f-aa3f-85f12087b2a3") : secret "dns-default-metrics-tls" not found Sep 17 23:06:47.427707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427388 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce53708a-abfa-4b74-b949-2456ce5cd297-ca-trust-extracted\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.427707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce53708a-abfa-4b74-b949-2456ce5cd297-trusted-ca\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.427707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427440 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-bound-sa-token\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.427707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427465 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrg9q\" (UniqueName: \"kubernetes.io/projected/d16d7679-5919-42cb-b4c7-379fadd35925-kube-api-access-wrg9q\") pod \"ingress-canary-6xlrg\" (UID: \"d16d7679-5919-42cb-b4c7-379fadd35925\") " pod="openshift-ingress-canary/ingress-canary-6xlrg" Sep 17 23:06:47.427707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427507 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce53708a-abfa-4b74-b949-2456ce5cd297-installation-pull-secrets\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.427707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjlqp\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-kube-api-access-sjlqp\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.427707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427626 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-certificates\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.427707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427637 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-tmp-dir\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.427707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427668 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d16d7679-5919-42cb-b4c7-379fadd35925-cert\") pod \"ingress-canary-6xlrg\" (UID: \"d16d7679-5919-42cb-b4c7-379fadd35925\") " pod="openshift-ingress-canary/ingress-canary-6xlrg" Sep 17 23:06:47.427707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427690 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-config-volume\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.428176 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427722 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnbvt\" (UniqueName: \"kubernetes.io/projected/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-kube-api-access-qnbvt\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.428176 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.427793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce53708a-abfa-4b74-b949-2456ce5cd297-ca-trust-extracted\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.429139 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.429108 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-config-volume\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.429800 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.429386 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce53708a-abfa-4b74-b949-2456ce5cd297-trusted-ca\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.429800 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.429722 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-certificates\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.432868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.432545 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce53708a-abfa-4b74-b949-2456ce5cd297-image-registry-private-configuration\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.433018 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.432888 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce53708a-abfa-4b74-b949-2456ce5cd297-installation-pull-secrets\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.434445 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.434403 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d16d7679-5919-42cb-b4c7-379fadd35925-cert\") pod \"ingress-canary-6xlrg\" (UID: \"d16d7679-5919-42cb-b4c7-379fadd35925\") " pod="openshift-ingress-canary/ingress-canary-6xlrg" Sep 17 23:06:47.440163 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.440143 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-bound-sa-token\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.441564 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.441539 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrg9q\" (UniqueName: \"kubernetes.io/projected/d16d7679-5919-42cb-b4c7-379fadd35925-kube-api-access-wrg9q\") pod \"ingress-canary-6xlrg\" (UID: \"d16d7679-5919-42cb-b4c7-379fadd35925\") " pod="openshift-ingress-canary/ingress-canary-6xlrg" Sep 17 23:06:47.443234 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.443191 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjlqp\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-kube-api-access-sjlqp\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.443433 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.443397 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnbvt\" (UniqueName: \"kubernetes.io/projected/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-kube-api-access-qnbvt\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.450746 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.450717 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:47.511926 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.511882 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6xlrg" Sep 17 23:06:47.551178 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.551141 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:47.802302 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.802232 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6xlrg"] Sep 17 23:06:47.807493 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:06:47.807458 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16d7679_5919_42cb_b4c7_379fadd35925.slice/crio-fb8c0fb37bd792bf7e5304579e3d5557835d67246bda3bc31389def863f71c71 WatchSource:0}: Error finding container fb8c0fb37bd792bf7e5304579e3d5557835d67246bda3bc31389def863f71c71: Status 404 returned error can't find the container with id fb8c0fb37bd792bf7e5304579e3d5557835d67246bda3bc31389def863f71c71 Sep 17 23:06:47.932916 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.932809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:47.932916 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:47.932857 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:47.933397 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:47.932991 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 23:06:47.933397 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:47.933007 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f64f764df-gtws9: secret "image-registry-tls" not found Sep 17 23:06:47.933397 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:47.932991 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:06:47.933397 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:47.933059 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls podName:ce53708a-abfa-4b74-b949-2456ce5cd297 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:48.933044597 +0000 UTC m=+37.050524025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls") pod "image-registry-7f64f764df-gtws9" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297") : secret "image-registry-tls" not found Sep 17 23:06:47.933397 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:47.933109 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls podName:5b7ac966-8594-4a9f-aa3f-85f12087b2a3 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:48.933090537 +0000 UTC m=+37.050569979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls") pod "dns-default-2fdkb" (UID: "5b7ac966-8594-4a9f-aa3f-85f12087b2a3") : secret "dns-default-metrics-tls" not found Sep 17 23:06:48.281009 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:48.280976 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:06:48.281190 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:48.280976 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:06:48.283624 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:48.283595 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 23:06:48.284372 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:48.284334 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 23:06:48.284372 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:48.284343 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t46xp\"" Sep 17 23:06:48.552320 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:48.552221 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6xlrg" event={"ID":"d16d7679-5919-42cb-b4c7-379fadd35925","Type":"ContainerStarted","Data":"fb8c0fb37bd792bf7e5304579e3d5557835d67246bda3bc31389def863f71c71"} Sep 17 23:06:48.940946 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:48.940844 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:48.940946 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:48.940896 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:48.941431 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:48.941018 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:06:48.941431 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:48.941021 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 23:06:48.941431 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:48.941092 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls podName:5b7ac966-8594-4a9f-aa3f-85f12087b2a3 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:50.9410765 +0000 UTC m=+39.058555946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls") pod "dns-default-2fdkb" (UID: "5b7ac966-8594-4a9f-aa3f-85f12087b2a3") : secret "dns-default-metrics-tls" not found Sep 17 23:06:48.941431 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:48.941096 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f64f764df-gtws9: secret "image-registry-tls" not found Sep 17 23:06:48.941431 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:48.941146 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls podName:ce53708a-abfa-4b74-b949-2456ce5cd297 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:50.941131183 +0000 UTC m=+39.058610632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls") pod "image-registry-7f64f764df-gtws9" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297") : secret "image-registry-tls" not found Sep 17 23:06:49.091913 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.091826 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-sx67h"] Sep 17 23:06:49.097756 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.097732 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-sx67h" Sep 17 23:06:49.100765 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.100641 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 17 23:06:49.100931 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.100766 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5xvzv\"" Sep 17 23:06:49.100931 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.100641 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 17 23:06:49.104589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.104529 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-sx67h"] Sep 17 23:06:49.111985 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.111961 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj"] Sep 17 23:06:49.116594 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.116570 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv"] Sep 17 23:06:49.116847 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.116828 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:49.120109 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.119874 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 17 23:06:49.120109 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.119921 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 23:06:49.120109 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.120018 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 23:06:49.120313 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.120256 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-kfrv8"] Sep 17 23:06:49.120430 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.120412 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:06:49.121768 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.121629 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-64ksw\"" Sep 17 23:06:49.121871 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.121798 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 17 23:06:49.122806 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.122741 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h5922\"" Sep 17 23:06:49.123303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.123065 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 17 23:06:49.123303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.123164 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 17 23:06:49.123303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.123302 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 17 23:06:49.125449 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.125429 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv"] Sep 17 23:06:49.125568 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.125534 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.129312 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.129293 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 17 23:06:49.129709 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.129440 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-kfrv8"] Sep 17 23:06:49.129709 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.129540 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 17 23:06:49.129709 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.129614 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 17 23:06:49.129709 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.129666 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 17 23:06:49.130164 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.129670 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-r97p5\"" Sep 17 23:06:49.131197 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.131176 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj"] Sep 17 23:06:49.134585 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.134536 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 17 23:06:49.196536 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.196479 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns"] Sep 17 23:06:49.207015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.206984 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-gw259"] Sep 17 23:06:49.207207 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.207149 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" Sep 17 23:06:49.211669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.211276 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 17 23:06:49.211669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.211299 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gmtd8\"" Sep 17 23:06:49.211669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.211503 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 17 23:06:49.211669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.211585 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 17 23:06:49.211669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.211706 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 17 23:06:49.215785 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.214747 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-5bgrm"] Sep 17 23:06:49.215785 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.214845 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.219920 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.219692 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 17 23:06:49.220040 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.219981 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5vtw6\"" Sep 17 23:06:49.220416 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.220106 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 23:06:49.220416 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.220175 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 17 23:06:49.220416 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.220258 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 23:06:49.225025 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.225001 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 17 23:06:49.226085 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.226062 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg"] Sep 17 23:06:49.226464 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.226446 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-5bgrm" Sep 17 23:06:49.229559 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.229533 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6v48z\"" Sep 17 23:06:49.233272 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.233249 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns"] Sep 17 23:06:49.233436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.233292 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg"] Sep 17 23:06:49.233436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.233315 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-gw259"] Sep 17 23:06:49.233436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.233335 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-5bgrm"] Sep 17 23:06:49.233436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.233371 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" Sep 17 23:06:49.235447 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.235426 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 17 23:06:49.235543 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.235447 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 17 23:06:49.235706 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.235685 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 17 23:06:49.235920 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.235718 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 17 23:06:49.236095 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.235783 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2gkdc\"" Sep 17 23:06:49.242828 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.242806 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c69cad18-c083-405a-aed3-91740cc79fd4-config\") pod \"console-operator-fd4bb46c-kfrv8\" (UID: \"c69cad18-c083-405a-aed3-91740cc79fd4\") " pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.242934 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.242913 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wgq8\" (UniqueName: \"kubernetes.io/projected/c69cad18-c083-405a-aed3-91740cc79fd4-kube-api-access-4wgq8\") pod \"console-operator-fd4bb46c-kfrv8\" (UID: \"c69cad18-c083-405a-aed3-91740cc79fd4\") " pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.243004 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.242947 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:49.243077 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.243047 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c69cad18-c083-405a-aed3-91740cc79fd4-serving-cert\") pod \"console-operator-fd4bb46c-kfrv8\" (UID: \"c69cad18-c083-405a-aed3-91740cc79fd4\") " pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.243123 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.243096 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljmdp\" (UniqueName: \"kubernetes.io/projected/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-kube-api-access-ljmdp\") pod \"cluster-samples-operator-685d7d68b5-lrvzv\" (UID: \"c27493ad-ac0b-46cb-a7bd-92da0fb886bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:06:49.243171 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.243120 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c69cad18-c083-405a-aed3-91740cc79fd4-trusted-ca\") pod \"console-operator-fd4bb46c-kfrv8\" (UID: \"c69cad18-c083-405a-aed3-91740cc79fd4\") " pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.243171 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.243143 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:49.243261 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.243179 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pp74\" (UniqueName: \"kubernetes.io/projected/97e32c44-0147-4b12-ac41-5f6643cbd98f-kube-api-access-5pp74\") pod \"volume-data-source-validator-5c4db8bdf9-sx67h\" (UID: \"97e32c44-0147-4b12-ac41-5f6643cbd98f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-sx67h" Sep 17 23:06:49.243261 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.243208 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-lrvzv\" (UID: \"c27493ad-ac0b-46cb-a7bd-92da0fb886bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:06:49.243261 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.243248 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjh52\" (UniqueName: \"kubernetes.io/projected/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-kube-api-access-fjh52\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:49.299085 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.299052 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5cd8bf6458-jhsss"] Sep 17 23:06:49.305020 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.304993 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.307805 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.307782 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 17 23:06:49.307958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.307851 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 23:06:49.307958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.307941 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 17 23:06:49.308073 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.307856 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 17 23:06:49.308073 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.308044 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vv779\"" Sep 17 23:06:49.308166 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.308136 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 23:06:49.308216 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.308168 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 17 23:06:49.312280 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.312255 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cd8bf6458-jhsss"] Sep 17 23:06:49.344556 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344521 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmpfq\" (UniqueName: \"kubernetes.io/projected/621bb439-2d17-4a60-877f-813e7a70be1c-kube-api-access-lmpfq\") pod \"network-check-source-7cbc45fddc-5bgrm\" (UID: \"621bb439-2d17-4a60-877f-813e7a70be1c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-5bgrm" Sep 17 23:06:49.344736 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344562 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38d9d121-59fe-483b-a6c4-f29510ab8230-config\") pod \"service-ca-operator-57cdc4b768-g6ngg\" (UID: \"38d9d121-59fe-483b-a6c4-f29510ab8230\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" Sep 17 23:06:49.344736 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344605 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/555eaa00-e5cb-49e7-adbb-18ccfa293242-snapshots\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.344736 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344640 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phzzq\" (UniqueName: \"kubernetes.io/projected/a9479c5a-db24-4df7-9402-b6dffa701fa4-kube-api-access-phzzq\") pod \"kube-storage-version-migrator-operator-55889f7b47-4gmns\" (UID: \"a9479c5a-db24-4df7-9402-b6dffa701fa4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" Sep 17 23:06:49.344736 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344706 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wgq8\" (UniqueName: \"kubernetes.io/projected/c69cad18-c083-405a-aed3-91740cc79fd4-kube-api-access-4wgq8\") pod \"console-operator-fd4bb46c-kfrv8\" (UID: \"c69cad18-c083-405a-aed3-91740cc79fd4\") " pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.344736 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344732 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxz8w\" (UniqueName: \"kubernetes.io/projected/38d9d121-59fe-483b-a6c4-f29510ab8230-kube-api-access-xxz8w\") pod \"service-ca-operator-57cdc4b768-g6ngg\" (UID: \"38d9d121-59fe-483b-a6c4-f29510ab8230\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" Sep 17 23:06:49.345119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344768 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38d9d121-59fe-483b-a6c4-f29510ab8230-serving-cert\") pod \"service-ca-operator-57cdc4b768-g6ngg\" (UID: \"38d9d121-59fe-483b-a6c4-f29510ab8230\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" Sep 17 23:06:49.345119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344806 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:49.345119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/555eaa00-e5cb-49e7-adbb-18ccfa293242-serving-cert\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.345119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c69cad18-c083-405a-aed3-91740cc79fd4-serving-cert\") pod \"console-operator-fd4bb46c-kfrv8\" (UID: \"c69cad18-c083-405a-aed3-91740cc79fd4\") " pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.345119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344881 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/555eaa00-e5cb-49e7-adbb-18ccfa293242-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.345119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-lrvzv\" (UID: \"c27493ad-ac0b-46cb-a7bd-92da0fb886bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:06:49.345119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344948 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjh52\" (UniqueName: \"kubernetes.io/projected/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-kube-api-access-fjh52\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:49.345119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.344978 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/555eaa00-e5cb-49e7-adbb-18ccfa293242-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.345119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.345018 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9479c5a-db24-4df7-9402-b6dffa701fa4-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-4gmns\" (UID: \"a9479c5a-db24-4df7-9402-b6dffa701fa4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" Sep 17 23:06:49.345119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.345049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c69cad18-c083-405a-aed3-91740cc79fd4-config\") pod \"console-operator-fd4bb46c-kfrv8\" (UID: \"c69cad18-c083-405a-aed3-91740cc79fd4\") " pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.345119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.345091 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:49.345763 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.345129 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qswjv\" (UniqueName: \"kubernetes.io/projected/555eaa00-e5cb-49e7-adbb-18ccfa293242-kube-api-access-qswjv\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.345763 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.345160 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c69cad18-c083-405a-aed3-91740cc79fd4-trusted-ca\") pod \"console-operator-fd4bb46c-kfrv8\" (UID: \"c69cad18-c083-405a-aed3-91740cc79fd4\") " pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.345763 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.345180 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/555eaa00-e5cb-49e7-adbb-18ccfa293242-tmp\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.345763 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.345206 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljmdp\" (UniqueName: \"kubernetes.io/projected/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-kube-api-access-ljmdp\") pod \"cluster-samples-operator-685d7d68b5-lrvzv\" (UID: \"c27493ad-ac0b-46cb-a7bd-92da0fb886bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:06:49.345763 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.345238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9479c5a-db24-4df7-9402-b6dffa701fa4-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-4gmns\" (UID: \"a9479c5a-db24-4df7-9402-b6dffa701fa4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" Sep 17 23:06:49.345763 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.345266 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pp74\" (UniqueName: \"kubernetes.io/projected/97e32c44-0147-4b12-ac41-5f6643cbd98f-kube-api-access-5pp74\") pod \"volume-data-source-validator-5c4db8bdf9-sx67h\" (UID: \"97e32c44-0147-4b12-ac41-5f6643cbd98f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-sx67h" Sep 17 23:06:49.345763 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:49.345734 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 23:06:49.346109 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:49.345799 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls podName:c27493ad-ac0b-46cb-a7bd-92da0fb886bc nodeName:}" failed. No retries permitted until 2026-09-17 23:06:49.845778259 +0000 UTC m=+37.963257688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-lrvzv" (UID: "c27493ad-ac0b-46cb-a7bd-92da0fb886bc") : secret "samples-operator-tls" not found Sep 17 23:06:49.346109 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:49.346009 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 23:06:49.346109 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:49.346056 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls podName:b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:49.846041607 +0000 UTC m=+37.963521038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-r2vwj" (UID: "b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5") : secret "cluster-monitoring-operator-tls" not found Sep 17 23:06:49.346391 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.346370 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c69cad18-c083-405a-aed3-91740cc79fd4-config\") pod \"console-operator-fd4bb46c-kfrv8\" (UID: \"c69cad18-c083-405a-aed3-91740cc79fd4\") " pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.346447 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.346370 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:49.346584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.346562 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c69cad18-c083-405a-aed3-91740cc79fd4-trusted-ca\") pod \"console-operator-fd4bb46c-kfrv8\" (UID: \"c69cad18-c083-405a-aed3-91740cc79fd4\") " pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.349755 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.349716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c69cad18-c083-405a-aed3-91740cc79fd4-serving-cert\") pod \"console-operator-fd4bb46c-kfrv8\" (UID: \"c69cad18-c083-405a-aed3-91740cc79fd4\") " pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.353795 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.353751 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wgq8\" (UniqueName: \"kubernetes.io/projected/c69cad18-c083-405a-aed3-91740cc79fd4-kube-api-access-4wgq8\") pod \"console-operator-fd4bb46c-kfrv8\" (UID: \"c69cad18-c083-405a-aed3-91740cc79fd4\") " pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.355266 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.354815 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pp74\" (UniqueName: \"kubernetes.io/projected/97e32c44-0147-4b12-ac41-5f6643cbd98f-kube-api-access-5pp74\") pod \"volume-data-source-validator-5c4db8bdf9-sx67h\" (UID: \"97e32c44-0147-4b12-ac41-5f6643cbd98f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-sx67h" Sep 17 23:06:49.355266 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.354827 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljmdp\" (UniqueName: \"kubernetes.io/projected/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-kube-api-access-ljmdp\") pod \"cluster-samples-operator-685d7d68b5-lrvzv\" (UID: \"c27493ad-ac0b-46cb-a7bd-92da0fb886bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:06:49.355266 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.355228 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjh52\" (UniqueName: \"kubernetes.io/projected/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-kube-api-access-fjh52\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:49.400095 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.400022 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:49.409850 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.409819 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-sx67h" Sep 17 23:06:49.445727 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.445695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9479c5a-db24-4df7-9402-b6dffa701fa4-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-4gmns\" (UID: \"a9479c5a-db24-4df7-9402-b6dffa701fa4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" Sep 17 23:06:49.445891 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.445753 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.445891 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.445798 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qswjv\" (UniqueName: \"kubernetes.io/projected/555eaa00-e5cb-49e7-adbb-18ccfa293242-kube-api-access-qswjv\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.445891 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.445823 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-default-certificate\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.445891 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.445853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/555eaa00-e5cb-49e7-adbb-18ccfa293242-tmp\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.446137 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.445926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9479c5a-db24-4df7-9402-b6dffa701fa4-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-4gmns\" (UID: \"a9479c5a-db24-4df7-9402-b6dffa701fa4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" Sep 17 23:06:49.446137 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.445980 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.446137 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.446016 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmpfq\" (UniqueName: \"kubernetes.io/projected/621bb439-2d17-4a60-877f-813e7a70be1c-kube-api-access-lmpfq\") pod \"network-check-source-7cbc45fddc-5bgrm\" (UID: \"621bb439-2d17-4a60-877f-813e7a70be1c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-5bgrm" Sep 17 23:06:49.446137 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.446035 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38d9d121-59fe-483b-a6c4-f29510ab8230-config\") pod \"service-ca-operator-57cdc4b768-g6ngg\" (UID: \"38d9d121-59fe-483b-a6c4-f29510ab8230\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" Sep 17 23:06:49.446137 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.446052 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-stats-auth\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.446137 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.446075 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fskzb\" (UniqueName: \"kubernetes.io/projected/0938e02b-9bf2-4f98-92b4-ee86260aef8f-kube-api-access-fskzb\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.446137 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.446100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/555eaa00-e5cb-49e7-adbb-18ccfa293242-snapshots\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.446137 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.446125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzzq\" (UniqueName: \"kubernetes.io/projected/a9479c5a-db24-4df7-9402-b6dffa701fa4-kube-api-access-phzzq\") pod \"kube-storage-version-migrator-operator-55889f7b47-4gmns\" (UID: \"a9479c5a-db24-4df7-9402-b6dffa701fa4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" Sep 17 23:06:49.446608 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.446183 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxz8w\" (UniqueName: \"kubernetes.io/projected/38d9d121-59fe-483b-a6c4-f29510ab8230-kube-api-access-xxz8w\") pod \"service-ca-operator-57cdc4b768-g6ngg\" (UID: \"38d9d121-59fe-483b-a6c4-f29510ab8230\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" Sep 17 23:06:49.446965 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.446199 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/555eaa00-e5cb-49e7-adbb-18ccfa293242-tmp\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.448509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.447497 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9479c5a-db24-4df7-9402-b6dffa701fa4-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-4gmns\" (UID: \"a9479c5a-db24-4df7-9402-b6dffa701fa4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" Sep 17 23:06:49.448509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.447719 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38d9d121-59fe-483b-a6c4-f29510ab8230-serving-cert\") pod \"service-ca-operator-57cdc4b768-g6ngg\" (UID: \"38d9d121-59fe-483b-a6c4-f29510ab8230\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" Sep 17 23:06:49.448509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.447892 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/555eaa00-e5cb-49e7-adbb-18ccfa293242-snapshots\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.448509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.448007 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38d9d121-59fe-483b-a6c4-f29510ab8230-config\") pod \"service-ca-operator-57cdc4b768-g6ngg\" (UID: \"38d9d121-59fe-483b-a6c4-f29510ab8230\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" Sep 17 23:06:49.448509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.448026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/555eaa00-e5cb-49e7-adbb-18ccfa293242-serving-cert\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.448509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.448084 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/555eaa00-e5cb-49e7-adbb-18ccfa293242-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.448509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.448143 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/555eaa00-e5cb-49e7-adbb-18ccfa293242-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.448873 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.448807 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/555eaa00-e5cb-49e7-adbb-18ccfa293242-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.449451 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.449431 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:49.451442 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.451423 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/555eaa00-e5cb-49e7-adbb-18ccfa293242-serving-cert\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.452311 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.452283 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/555eaa00-e5cb-49e7-adbb-18ccfa293242-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.453256 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.453232 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9479c5a-db24-4df7-9402-b6dffa701fa4-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-4gmns\" (UID: \"a9479c5a-db24-4df7-9402-b6dffa701fa4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" Sep 17 23:06:49.454560 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.454531 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38d9d121-59fe-483b-a6c4-f29510ab8230-serving-cert\") pod \"service-ca-operator-57cdc4b768-g6ngg\" (UID: \"38d9d121-59fe-483b-a6c4-f29510ab8230\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" Sep 17 23:06:49.456234 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.456192 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qswjv\" (UniqueName: \"kubernetes.io/projected/555eaa00-e5cb-49e7-adbb-18ccfa293242-kube-api-access-qswjv\") pod \"insights-operator-6f9c644dd9-gw259\" (UID: \"555eaa00-e5cb-49e7-adbb-18ccfa293242\") " pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.456353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.456331 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmpfq\" (UniqueName: \"kubernetes.io/projected/621bb439-2d17-4a60-877f-813e7a70be1c-kube-api-access-lmpfq\") pod \"network-check-source-7cbc45fddc-5bgrm\" (UID: \"621bb439-2d17-4a60-877f-813e7a70be1c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-5bgrm" Sep 17 23:06:49.456810 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.456788 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxz8w\" (UniqueName: \"kubernetes.io/projected/38d9d121-59fe-483b-a6c4-f29510ab8230-kube-api-access-xxz8w\") pod \"service-ca-operator-57cdc4b768-g6ngg\" (UID: \"38d9d121-59fe-483b-a6c4-f29510ab8230\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" Sep 17 23:06:49.457153 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.457132 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:49.468236 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.468158 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phzzq\" (UniqueName: \"kubernetes.io/projected/a9479c5a-db24-4df7-9402-b6dffa701fa4-kube-api-access-phzzq\") pod \"kube-storage-version-migrator-operator-55889f7b47-4gmns\" (UID: \"a9479c5a-db24-4df7-9402-b6dffa701fa4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" Sep 17 23:06:49.524481 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.524435 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" Sep 17 23:06:49.534690 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.534622 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-gw259" Sep 17 23:06:49.542145 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.542122 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-5bgrm" Sep 17 23:06:49.548970 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.548944 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" Sep 17 23:06:49.549206 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.549185 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-stats-auth\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.549271 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.549227 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fskzb\" (UniqueName: \"kubernetes.io/projected/0938e02b-9bf2-4f98-92b4-ee86260aef8f-kube-api-access-fskzb\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.549489 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.549374 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.549489 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.549412 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-default-certificate\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.549489 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.549451 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.549695 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:49.549550 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 23:06:49.549695 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:49.549611 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs podName:0938e02b-9bf2-4f98-92b4-ee86260aef8f nodeName:}" failed. No retries permitted until 2026-09-17 23:06:50.049589312 +0000 UTC m=+38.167068755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs") pod "router-default-5cd8bf6458-jhsss" (UID: "0938e02b-9bf2-4f98-92b4-ee86260aef8f") : secret "router-metrics-certs-default" not found Sep 17 23:06:49.549816 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:49.549701 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle podName:0938e02b-9bf2-4f98-92b4-ee86260aef8f nodeName:}" failed. No retries permitted until 2026-09-17 23:06:50.049687111 +0000 UTC m=+38.167166563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle") pod "router-default-5cd8bf6458-jhsss" (UID: "0938e02b-9bf2-4f98-92b4-ee86260aef8f") : configmap references non-existent config key: service-ca.crt Sep 17 23:06:49.551996 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.551974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-stats-auth\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.552306 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.552258 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-default-certificate\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.558698 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.558666 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fskzb\" (UniqueName: \"kubernetes.io/projected/0938e02b-9bf2-4f98-92b4-ee86260aef8f-kube-api-access-fskzb\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:49.852884 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.852793 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:49.852884 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:49.852848 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-lrvzv\" (UID: \"c27493ad-ac0b-46cb-a7bd-92da0fb886bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:06:49.853133 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:49.852971 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 23:06:49.853133 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:49.852977 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 23:06:49.853133 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:49.853034 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls podName:c27493ad-ac0b-46cb-a7bd-92da0fb886bc nodeName:}" failed. No retries permitted until 2026-09-17 23:06:50.853015897 +0000 UTC m=+38.970495327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-lrvzv" (UID: "c27493ad-ac0b-46cb-a7bd-92da0fb886bc") : secret "samples-operator-tls" not found Sep 17 23:06:49.853133 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:49.853099 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls podName:b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:50.853082743 +0000 UTC m=+38.970562172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-r2vwj" (UID: "b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5") : secret "cluster-monitoring-operator-tls" not found Sep 17 23:06:50.054254 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:50.054217 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:50.054917 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:50.054288 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:50.054917 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:50.054379 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle podName:0938e02b-9bf2-4f98-92b4-ee86260aef8f nodeName:}" failed. No retries permitted until 2026-09-17 23:06:51.054360643 +0000 UTC m=+39.171840094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle") pod "router-default-5cd8bf6458-jhsss" (UID: "0938e02b-9bf2-4f98-92b4-ee86260aef8f") : configmap references non-existent config key: service-ca.crt Sep 17 23:06:50.054917 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:50.054408 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 23:06:50.054917 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:50.054471 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs podName:0938e02b-9bf2-4f98-92b4-ee86260aef8f nodeName:}" failed. No retries permitted until 2026-09-17 23:06:51.054454466 +0000 UTC m=+39.171933913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs") pod "router-default-5cd8bf6458-jhsss" (UID: "0938e02b-9bf2-4f98-92b4-ee86260aef8f") : secret "router-metrics-certs-default" not found Sep 17 23:06:50.862420 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:50.862328 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:50.862420 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:50.862385 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-lrvzv\" (UID: \"c27493ad-ac0b-46cb-a7bd-92da0fb886bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:06:50.862759 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:50.862496 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 23:06:50.862759 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:50.862565 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls podName:b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:52.862549507 +0000 UTC m=+40.980028950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-r2vwj" (UID: "b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5") : secret "cluster-monitoring-operator-tls" not found Sep 17 23:06:50.862759 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:50.862494 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 23:06:50.862759 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:50.862653 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls podName:c27493ad-ac0b-46cb-a7bd-92da0fb886bc nodeName:}" failed. No retries permitted until 2026-09-17 23:06:52.862638149 +0000 UTC m=+40.980117591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-lrvzv" (UID: "c27493ad-ac0b-46cb-a7bd-92da0fb886bc") : secret "samples-operator-tls" not found Sep 17 23:06:50.963496 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:50.963443 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:50.963681 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:50.963504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:50.963681 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:50.963612 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:06:50.963799 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:50.963694 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls podName:5b7ac966-8594-4a9f-aa3f-85f12087b2a3 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:54.963674317 +0000 UTC m=+43.081153763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls") pod "dns-default-2fdkb" (UID: "5b7ac966-8594-4a9f-aa3f-85f12087b2a3") : secret "dns-default-metrics-tls" not found Sep 17 23:06:50.963799 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:50.963725 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 23:06:50.963799 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:50.963747 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f64f764df-gtws9: secret "image-registry-tls" not found Sep 17 23:06:50.963983 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:50.963804 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls podName:ce53708a-abfa-4b74-b949-2456ce5cd297 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:54.963788127 +0000 UTC m=+43.081267573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls") pod "image-registry-7f64f764df-gtws9" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297") : secret "image-registry-tls" not found Sep 17 23:06:51.064861 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:51.064817 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:51.065337 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:51.065016 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 23:06:51.065337 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:51.065028 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:51.065337 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:51.065100 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs podName:0938e02b-9bf2-4f98-92b4-ee86260aef8f nodeName:}" failed. No retries permitted until 2026-09-17 23:06:53.065080496 +0000 UTC m=+41.182559945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs") pod "router-default-5cd8bf6458-jhsss" (UID: "0938e02b-9bf2-4f98-92b4-ee86260aef8f") : secret "router-metrics-certs-default" not found Sep 17 23:06:51.065337 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:51.065190 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle podName:0938e02b-9bf2-4f98-92b4-ee86260aef8f nodeName:}" failed. No retries permitted until 2026-09-17 23:06:53.065164216 +0000 UTC m=+41.182643660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle") pod "router-default-5cd8bf6458-jhsss" (UID: "0938e02b-9bf2-4f98-92b4-ee86260aef8f") : configmap references non-existent config key: service-ca.crt Sep 17 23:06:51.751242 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:51.750937 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns"] Sep 17 23:06:51.762145 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:51.762110 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-sx67h"] Sep 17 23:06:51.764829 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:51.764805 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-5bgrm"] Sep 17 23:06:51.765686 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:51.765667 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg"] Sep 17 23:06:51.767670 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:51.767650 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-gw259"] Sep 17 23:06:51.776929 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:51.776329 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-kfrv8"] Sep 17 23:06:51.833998 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:51.833972 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:51.896767 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:51.896738 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:52.571807 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.571768 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-5bgrm" event={"ID":"621bb439-2d17-4a60-877f-813e7a70be1c","Type":"ContainerStarted","Data":"edae0841e6666ef074ed908f267efdf0277325b45bb42edb94aff3147412b5f8"} Sep 17 23:06:52.574194 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.574162 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" event={"ID":"c69cad18-c083-405a-aed3-91740cc79fd4","Type":"ContainerStarted","Data":"556adfcce87b5f52726a4822ae322dc63071f3c03352b9fea9a7070444288941"} Sep 17 23:06:52.580482 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.580434 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-gw259" event={"ID":"555eaa00-e5cb-49e7-adbb-18ccfa293242","Type":"ContainerStarted","Data":"01b62fd015bb85791ea25e08dbe87635b663df4e16ff785339e1f0173bf04a86"} Sep 17 23:06:52.589453 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.589420 2559 generic.go:356] "Generic (PLEG): container finished" podID="4cd6491b-4078-4652-98cc-31a2b75c6911" containerID="8260555f41226d30f0bdf00de4a81b296f8c4078af891313f29bacab022a7151" exitCode=0 Sep 17 23:06:52.589566 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.589510 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx2wl" event={"ID":"4cd6491b-4078-4652-98cc-31a2b75c6911","Type":"ContainerDied","Data":"8260555f41226d30f0bdf00de4a81b296f8c4078af891313f29bacab022a7151"} Sep 17 23:06:52.596755 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.596662 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" event={"ID":"38d9d121-59fe-483b-a6c4-f29510ab8230","Type":"ContainerStarted","Data":"67c122bdd1424f6dfd6c8306a80e2f9681cef7c6360bbf7d0a7f7b702b05561b"} Sep 17 23:06:52.598782 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.598682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" event={"ID":"a9479c5a-db24-4df7-9402-b6dffa701fa4","Type":"ContainerStarted","Data":"9ab13069b06ce9469f42b3e2edb2fe65eb50f17792e3ebfa6cffa471120bdd72"} Sep 17 23:06:52.603946 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.603414 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6xlrg" event={"ID":"d16d7679-5919-42cb-b4c7-379fadd35925","Type":"ContainerStarted","Data":"2723980836a28dddd38b3a53d9af0e365442d0a3ef1306ff19906c6d1904d1e4"} Sep 17 23:06:52.606947 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.606921 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-sx67h" event={"ID":"97e32c44-0147-4b12-ac41-5f6643cbd98f","Type":"ContainerStarted","Data":"0e17ff9600c59e9beb4072fe968739f5b8b9c42e83a09b1972bb5451a2acee40"} Sep 17 23:06:52.639742 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.639391 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6xlrg" podStartSLOduration=1.993390093 podStartE2EDuration="5.63936929s" totalImagesPullingTime="3.645979197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:52.638363025 +0000 UTC m=+40.755842554" watchObservedRunningTime="2026-09-17 23:06:52.63936929 +0000 UTC m=+40.756848742" Sep 17 23:06:52.856946 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.856842 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:52.882720 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.882678 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:52.882945 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.882748 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-lrvzv\" (UID: \"c27493ad-ac0b-46cb-a7bd-92da0fb886bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:06:52.883063 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:52.883043 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 23:06:52.883168 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:52.883121 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls podName:b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:56.883099319 +0000 UTC m=+45.000578765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-r2vwj" (UID: "b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5") : secret "cluster-monitoring-operator-tls" not found Sep 17 23:06:52.883557 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:52.883539 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 23:06:52.883660 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:52.883604 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls podName:c27493ad-ac0b-46cb-a7bd-92da0fb886bc nodeName:}" failed. No retries permitted until 2026-09-17 23:06:56.883588505 +0000 UTC m=+45.001067948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-lrvzv" (UID: "c27493ad-ac0b-46cb-a7bd-92da0fb886bc") : secret "samples-operator-tls" not found Sep 17 23:06:52.954299 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:52.954250 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:53.084946 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.084879 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:53.085133 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.084968 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:53.085199 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:53.085183 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 23:06:53.085253 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:53.085247 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs podName:0938e02b-9bf2-4f98-92b4-ee86260aef8f nodeName:}" failed. No retries permitted until 2026-09-17 23:06:57.085228566 +0000 UTC m=+45.202708000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs") pod "router-default-5cd8bf6458-jhsss" (UID: "0938e02b-9bf2-4f98-92b4-ee86260aef8f") : secret "router-metrics-certs-default" not found Sep 17 23:06:53.085796 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:53.085767 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle podName:0938e02b-9bf2-4f98-92b4-ee86260aef8f nodeName:}" failed. No retries permitted until 2026-09-17 23:06:57.085726262 +0000 UTC m=+45.203205714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle") pod "router-default-5cd8bf6458-jhsss" (UID: "0938e02b-9bf2-4f98-92b4-ee86260aef8f") : configmap references non-existent config key: service-ca.crt Sep 17 23:06:53.573837 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.572643 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl"] Sep 17 23:06:53.583414 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.580944 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:06:53.583414 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.583313 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl"] Sep 17 23:06:53.583655 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.583545 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 17 23:06:53.583717 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.583705 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zhqw8\"" Sep 17 23:06:53.583934 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.583917 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 17 23:06:53.594590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.594436 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dp5fl\" (UID: \"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:06:53.594590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.594507 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-dp5fl\" (UID: \"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:06:53.615859 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.615666 2559 generic.go:356] "Generic (PLEG): container finished" podID="4cd6491b-4078-4652-98cc-31a2b75c6911" containerID="e665e8e894ab2033977f8532ccffa79d003e0c34f4dc2fa1d8bae986d938103f" exitCode=0 Sep 17 23:06:53.615859 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.615784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx2wl" event={"ID":"4cd6491b-4078-4652-98cc-31a2b75c6911","Type":"ContainerDied","Data":"e665e8e894ab2033977f8532ccffa79d003e0c34f4dc2fa1d8bae986d938103f"} Sep 17 23:06:53.696326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.695882 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dp5fl\" (UID: \"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:06:53.696326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.695969 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-dp5fl\" (UID: \"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:06:53.698984 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:53.697356 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 23:06:53.698984 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:53.697427 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert podName:8b5569f3-268e-4bd8-b7dc-6f5f0102f08c nodeName:}" failed. No retries permitted until 2026-09-17 23:06:54.197405964 +0000 UTC m=+42.314885408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-dp5fl" (UID: "8b5569f3-268e-4bd8-b7dc-6f5f0102f08c") : secret "networking-console-plugin-cert" not found Sep 17 23:06:53.698984 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.698334 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-dp5fl\" (UID: \"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:06:53.701929 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.701783 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:53.795779 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.795718 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:53.894680 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.894643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:53.958061 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:53.957930 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:54.200865 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:54.200829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dp5fl\" (UID: \"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:06:54.201071 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:54.201026 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 23:06:54.201140 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:54.201117 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert podName:8b5569f3-268e-4bd8-b7dc-6f5f0102f08c nodeName:}" failed. No retries permitted until 2026-09-17 23:06:55.201092086 +0000 UTC m=+43.318571536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-dp5fl" (UID: "8b5569f3-268e-4bd8-b7dc-6f5f0102f08c") : secret "networking-console-plugin-cert" not found Sep 17 23:06:54.628922 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:54.628818 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx2wl" event={"ID":"4cd6491b-4078-4652-98cc-31a2b75c6911","Type":"ContainerStarted","Data":"75015c49ec6bc8524d07fddf4b6413cf48d48b02cb3ed6bba27259db8864e12e"} Sep 17 23:06:54.630842 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:54.630798 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-sx67h" event={"ID":"97e32c44-0147-4b12-ac41-5f6643cbd98f","Type":"ContainerStarted","Data":"b556579588259540a18db566c162254e5280269307dd0ad250a9ee28eb47066e"} Sep 17 23:06:54.657633 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:54.657528 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lx2wl" podStartSLOduration=11.588634416 podStartE2EDuration="40.657514848s" totalImagesPullingTime="29.068880432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:54.656461755 +0000 UTC m=+42.773941234" watchObservedRunningTime="2026-09-17 23:06:54.657514848 +0000 UTC m=+42.774994299" Sep 17 23:06:54.670876 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:54.670718 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-sx67h" podStartSLOduration=3.939271147 podStartE2EDuration="5.670702859s" totalImagesPullingTime="1.731431712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:54.670597171 +0000 UTC m=+42.788076645" watchObservedRunningTime="2026-09-17 23:06:54.670702859 +0000 UTC m=+42.788182343" Sep 17 23:06:54.782641 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:54.782601 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:54.838575 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:54.838536 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:55.009161 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:55.009126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:06:55.009161 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:55.009165 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:06:55.009352 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:55.009282 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:06:55.009352 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:55.009298 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 23:06:55.009352 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:55.009311 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f64f764df-gtws9: secret "image-registry-tls" not found Sep 17 23:06:55.009352 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:55.009351 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls podName:5b7ac966-8594-4a9f-aa3f-85f12087b2a3 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:03.009335331 +0000 UTC m=+51.126814765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls") pod "dns-default-2fdkb" (UID: "5b7ac966-8594-4a9f-aa3f-85f12087b2a3") : secret "dns-default-metrics-tls" not found Sep 17 23:06:55.009484 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:55.009365 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls podName:ce53708a-abfa-4b74-b949-2456ce5cd297 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:03.009358998 +0000 UTC m=+51.126838427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls") pod "image-registry-7f64f764df-gtws9" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297") : secret "image-registry-tls" not found Sep 17 23:06:55.036325 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:55.036290 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:55.087453 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:55.087416 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:55.210778 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:55.210745 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dp5fl\" (UID: \"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:06:55.210968 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:55.210884 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 23:06:55.210968 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:55.210963 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert podName:8b5569f3-268e-4bd8-b7dc-6f5f0102f08c nodeName:}" failed. No retries permitted until 2026-09-17 23:06:57.210946324 +0000 UTC m=+45.328425776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-dp5fl" (UID: "8b5569f3-268e-4bd8-b7dc-6f5f0102f08c") : secret "networking-console-plugin-cert" not found Sep 17 23:06:56.731895 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:56.731863 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:56.923882 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:56.923840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-lrvzv\" (UID: \"c27493ad-ac0b-46cb-a7bd-92da0fb886bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:06:56.924092 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:56.924020 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 23:06:56.924092 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:56.924063 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:06:56.924092 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:56.924089 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls podName:c27493ad-ac0b-46cb-a7bd-92da0fb886bc nodeName:}" failed. No retries permitted until 2026-09-17 23:07:04.924070474 +0000 UTC m=+53.041549904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-lrvzv" (UID: "c27493ad-ac0b-46cb-a7bd-92da0fb886bc") : secret "samples-operator-tls" not found Sep 17 23:06:56.924260 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:56.924156 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 23:06:56.924260 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:56.924218 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls podName:b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:04.924200352 +0000 UTC m=+53.041679795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-r2vwj" (UID: "b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5") : secret "cluster-monitoring-operator-tls" not found Sep 17 23:06:57.126201 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:57.126112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:57.126201 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:57.126181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:06:57.126428 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:57.126300 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle podName:0938e02b-9bf2-4f98-92b4-ee86260aef8f nodeName:}" failed. No retries permitted until 2026-09-17 23:07:05.126281681 +0000 UTC m=+53.243761115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle") pod "router-default-5cd8bf6458-jhsss" (UID: "0938e02b-9bf2-4f98-92b4-ee86260aef8f") : configmap references non-existent config key: service-ca.crt Sep 17 23:06:57.126428 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:57.126302 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 23:06:57.126428 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:57.126356 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs podName:0938e02b-9bf2-4f98-92b4-ee86260aef8f nodeName:}" failed. No retries permitted until 2026-09-17 23:07:05.126340367 +0000 UTC m=+53.243819830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs") pod "router-default-5cd8bf6458-jhsss" (UID: "0938e02b-9bf2-4f98-92b4-ee86260aef8f") : secret "router-metrics-certs-default" not found Sep 17 23:06:57.226889 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:57.226844 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dp5fl\" (UID: \"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:06:57.227076 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:57.227026 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 23:06:57.227132 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:57.227109 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert podName:8b5569f3-268e-4bd8-b7dc-6f5f0102f08c nodeName:}" failed. No retries permitted until 2026-09-17 23:07:01.227087572 +0000 UTC m=+49.344567005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-dp5fl" (UID: "8b5569f3-268e-4bd8-b7dc-6f5f0102f08c") : secret "networking-console-plugin-cert" not found Sep 17 23:06:58.644233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:58.644137 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-5bgrm" event={"ID":"621bb439-2d17-4a60-877f-813e7a70be1c","Type":"ContainerStarted","Data":"e7b004ac1876d8a35f0205397b0a01bf405c9c4f87e119beeb3f930a63af99aa"} Sep 17 23:06:58.646048 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:58.646026 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/0.log" Sep 17 23:06:58.646191 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:58.646074 2559 generic.go:356] "Generic (PLEG): container finished" podID="c69cad18-c083-405a-aed3-91740cc79fd4" containerID="0ecd8ff00875a902b687daa7f39a159aa81d86b0e2ae1bede58c221d0dc61089" exitCode=255 Sep 17 23:06:58.646191 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:58.646166 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" event={"ID":"c69cad18-c083-405a-aed3-91740cc79fd4","Type":"ContainerDied","Data":"0ecd8ff00875a902b687daa7f39a159aa81d86b0e2ae1bede58c221d0dc61089"} Sep 17 23:06:58.646754 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:58.646735 2559 scope.go:117] "RemoveContainer" containerID="0ecd8ff00875a902b687daa7f39a159aa81d86b0e2ae1bede58c221d0dc61089" Sep 17 23:06:58.648266 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:58.648242 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-gw259" event={"ID":"555eaa00-e5cb-49e7-adbb-18ccfa293242","Type":"ContainerStarted","Data":"44902ceb92f25df18fedc8a3cb773d5a4c5b84d4175915d16b13c99be58e5560"} Sep 17 23:06:58.650355 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:58.650333 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" event={"ID":"38d9d121-59fe-483b-a6c4-f29510ab8230","Type":"ContainerStarted","Data":"e4ad10a0506bc34f35966a8ab17301555710342fa413d71800525276065d24b6"} Sep 17 23:06:58.652018 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:58.651997 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" event={"ID":"a9479c5a-db24-4df7-9402-b6dffa701fa4","Type":"ContainerStarted","Data":"20ab9191f293494f126fa22f76c679bebc68ffa34d174d04463e98b82f3bd8a3"} Sep 17 23:06:58.661079 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:58.660729 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-5bgrm" podStartSLOduration=3.314271732 podStartE2EDuration="9.660715601s" totalImagesPullingTime="6.346443869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:58.65975148 +0000 UTC m=+46.777230966" watchObservedRunningTime="2026-09-17 23:06:58.660715601 +0000 UTC m=+46.778195051" Sep 17 23:06:58.697982 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:58.697927 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" podStartSLOduration=3.363117367 podStartE2EDuration="9.697874123s" totalImagesPullingTime="6.334756756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:58.697440988 +0000 UTC m=+46.814920465" watchObservedRunningTime="2026-09-17 23:06:58.697874123 +0000 UTC m=+46.815353572" Sep 17 23:06:58.715819 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:58.715745 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" podStartSLOduration=3.374456333 podStartE2EDuration="9.715722476s" totalImagesPullingTime="6.341266143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:58.714276894 +0000 UTC m=+46.831756349" watchObservedRunningTime="2026-09-17 23:06:58.715722476 +0000 UTC m=+46.833201930" Sep 17 23:06:58.737000 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:58.736950 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-gw259" podStartSLOduration=3.39558542 podStartE2EDuration="9.736932587s" totalImagesPullingTime="6.341347167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:58.735530225 +0000 UTC m=+46.853009676" watchObservedRunningTime="2026-09-17 23:06:58.736932587 +0000 UTC m=+46.854412037" Sep 17 23:06:59.108350 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.108321 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:59.174992 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.174954 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:59.451803 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.451761 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:59.451993 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.451866 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:06:59.517557 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.517510 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:59.597622 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.597588 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vhnn4"] Sep 17 23:06:59.617047 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.617017 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:59.618672 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.618650 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vhnn4"] Sep 17 23:06:59.618814 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.618793 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.621273 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.621252 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 23:06:59.621390 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.621283 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ktswr\"" Sep 17 23:06:59.621562 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.621541 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 23:06:59.651330 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.651296 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4dxg\" (UniqueName: \"kubernetes.io/projected/22411ad6-db7e-412f-9088-4da16631a9cf-kube-api-access-j4dxg\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.651757 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.651363 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/22411ad6-db7e-412f-9088-4da16631a9cf-crio-socket\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.651757 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.651520 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.651757 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.651627 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/22411ad6-db7e-412f-9088-4da16631a9cf-data-volume\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.651757 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.651693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22411ad6-db7e-412f-9088-4da16631a9cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.657720 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.657680 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:06:59.658353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.658325 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/0.log" Sep 17 23:06:59.658473 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.658380 2559 generic.go:356] "Generic (PLEG): container finished" podID="c69cad18-c083-405a-aed3-91740cc79fd4" containerID="22a63b413aad3867ac243027ea85ee65433c94006150533fb5d9986cf0022041" exitCode=255 Sep 17 23:06:59.658543 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.658464 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" event={"ID":"c69cad18-c083-405a-aed3-91740cc79fd4","Type":"ContainerDied","Data":"22a63b413aad3867ac243027ea85ee65433c94006150533fb5d9986cf0022041"} Sep 17 23:06:59.658543 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.658532 2559 scope.go:117] "RemoveContainer" containerID="0ecd8ff00875a902b687daa7f39a159aa81d86b0e2ae1bede58c221d0dc61089" Sep 17 23:06:59.658889 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.658828 2559 scope.go:117] "RemoveContainer" containerID="22a63b413aad3867ac243027ea85ee65433c94006150533fb5d9986cf0022041" Sep 17 23:06:59.659225 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:59.659106 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-kfrv8_openshift-console-operator(c69cad18-c083-405a-aed3-91740cc79fd4)\"" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" podUID="c69cad18-c083-405a-aed3-91740cc79fd4" Sep 17 23:06:59.752428 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.752351 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.752578 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:59.752537 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 23:06:59.752654 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:59.752614 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls podName:22411ad6-db7e-412f-9088-4da16631a9cf nodeName:}" failed. No retries permitted until 2026-09-17 23:07:00.252593283 +0000 UTC m=+48.370072726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhnn4" (UID: "22411ad6-db7e-412f-9088-4da16631a9cf") : secret "insights-runtime-extractor-tls" not found Sep 17 23:06:59.752717 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.752675 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/22411ad6-db7e-412f-9088-4da16631a9cf-data-volume\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.752798 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.752775 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22411ad6-db7e-412f-9088-4da16631a9cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.752872 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.752825 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4dxg\" (UniqueName: \"kubernetes.io/projected/22411ad6-db7e-412f-9088-4da16631a9cf-kube-api-access-j4dxg\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.753011 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.752988 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/22411ad6-db7e-412f-9088-4da16631a9cf-crio-socket\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.753118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.753087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/22411ad6-db7e-412f-9088-4da16631a9cf-data-volume\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.753118 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:59.752995 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 23:06:59.753277 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:06:59.753262 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22411ad6-db7e-412f-9088-4da16631a9cf-kube-rbac-proxy-cm podName:22411ad6-db7e-412f-9088-4da16631a9cf nodeName:}" failed. No retries permitted until 2026-09-17 23:07:00.253244279 +0000 UTC m=+48.370723731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/22411ad6-db7e-412f-9088-4da16631a9cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhnn4" (UID: "22411ad6-db7e-412f-9088-4da16631a9cf") : configmap "kube-rbac-proxy" not found Sep 17 23:06:59.753349 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.753306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/22411ad6-db7e-412f-9088-4da16631a9cf-crio-socket\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.778240 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.778199 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4dxg\" (UniqueName: \"kubernetes.io/projected/22411ad6-db7e-412f-9088-4da16631a9cf-kube-api-access-j4dxg\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:06:59.892668 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.892628 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:06:59.949938 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.949880 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-r8t9z"] Sep 17 23:06:59.977780 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.977753 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-r8t9z"] Sep 17 23:06:59.977951 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.977866 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r8t9z" Sep 17 23:06:59.980328 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.980309 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 17 23:06:59.980328 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.980316 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 17 23:06:59.980560 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.980453 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-m2np6\"" Sep 17 23:06:59.997870 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:06:59.997831 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:00.055284 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:00.055201 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzb2t\" (UniqueName: \"kubernetes.io/projected/252f319b-a647-4f72-915c-2a4df27d83fe-kube-api-access-lzb2t\") pod \"migrator-76794cbd6c-r8t9z\" (UID: \"252f319b-a647-4f72-915c-2a4df27d83fe\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r8t9z" Sep 17 23:07:00.097836 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:00.097803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:00.156587 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:00.156556 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzb2t\" (UniqueName: \"kubernetes.io/projected/252f319b-a647-4f72-915c-2a4df27d83fe-kube-api-access-lzb2t\") pod \"migrator-76794cbd6c-r8t9z\" (UID: \"252f319b-a647-4f72-915c-2a4df27d83fe\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r8t9z" Sep 17 23:07:00.164248 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:00.164216 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzb2t\" (UniqueName: \"kubernetes.io/projected/252f319b-a647-4f72-915c-2a4df27d83fe-kube-api-access-lzb2t\") pod \"migrator-76794cbd6c-r8t9z\" (UID: \"252f319b-a647-4f72-915c-2a4df27d83fe\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r8t9z" Sep 17 23:07:00.257962 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:00.257924 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:07:00.258107 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:00.258027 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22411ad6-db7e-412f-9088-4da16631a9cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:07:00.258107 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:00.258081 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 23:07:00.258193 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:00.258137 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 23:07:00.258193 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:00.258155 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls podName:22411ad6-db7e-412f-9088-4da16631a9cf nodeName:}" failed. No retries permitted until 2026-09-17 23:07:01.25813888 +0000 UTC m=+49.375618326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhnn4" (UID: "22411ad6-db7e-412f-9088-4da16631a9cf") : secret "insights-runtime-extractor-tls" not found Sep 17 23:07:00.258193 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:00.258175 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22411ad6-db7e-412f-9088-4da16631a9cf-kube-rbac-proxy-cm podName:22411ad6-db7e-412f-9088-4da16631a9cf nodeName:}" failed. No retries permitted until 2026-09-17 23:07:01.25816483 +0000 UTC m=+49.375644260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/22411ad6-db7e-412f-9088-4da16631a9cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhnn4" (UID: "22411ad6-db7e-412f-9088-4da16631a9cf") : configmap "kube-rbac-proxy" not found Sep 17 23:07:00.287968 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:00.287941 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r8t9z" Sep 17 23:07:00.554029 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:00.554002 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fxvl4_d6070593-232f-4a38-b8e2-3380c4abe49e/dns-node-resolver/0.log" Sep 17 23:07:00.581429 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:00.581333 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-r8t9z"] Sep 17 23:07:00.586831 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:00.586802 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252f319b_a647_4f72_915c_2a4df27d83fe.slice/crio-7683d1ad3e62844894e2f1704ddbed92df0364d7167d9fcd1f5b0788ca59a4e4 WatchSource:0}: Error finding container 7683d1ad3e62844894e2f1704ddbed92df0364d7167d9fcd1f5b0788ca59a4e4: Status 404 returned error can't find the container with id 7683d1ad3e62844894e2f1704ddbed92df0364d7167d9fcd1f5b0788ca59a4e4 Sep 17 23:07:00.663017 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:00.662983 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:07:00.663452 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:00.663340 2559 scope.go:117] "RemoveContainer" containerID="22a63b413aad3867ac243027ea85ee65433c94006150533fb5d9986cf0022041" Sep 17 23:07:00.663537 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:00.663512 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-kfrv8_openshift-console-operator(c69cad18-c083-405a-aed3-91740cc79fd4)\"" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" podUID="c69cad18-c083-405a-aed3-91740cc79fd4" Sep 17 23:07:00.664258 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:00.664235 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r8t9z" event={"ID":"252f319b-a647-4f72-915c-2a4df27d83fe","Type":"ContainerStarted","Data":"7683d1ad3e62844894e2f1704ddbed92df0364d7167d9fcd1f5b0788ca59a4e4"} Sep 17 23:07:01.267915 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.267854 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:07:01.268112 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.268015 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22411ad6-db7e-412f-9088-4da16631a9cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:07:01.268112 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:01.268036 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 23:07:01.268268 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:01.268117 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls podName:22411ad6-db7e-412f-9088-4da16631a9cf nodeName:}" failed. No retries permitted until 2026-09-17 23:07:03.268094715 +0000 UTC m=+51.385574170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhnn4" (UID: "22411ad6-db7e-412f-9088-4da16631a9cf") : secret "insights-runtime-extractor-tls" not found Sep 17 23:07:01.268268 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:01.268147 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 23:07:01.268268 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.268177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dp5fl\" (UID: \"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:07:01.268268 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:01.268225 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22411ad6-db7e-412f-9088-4da16631a9cf-kube-rbac-proxy-cm podName:22411ad6-db7e-412f-9088-4da16631a9cf nodeName:}" failed. No retries permitted until 2026-09-17 23:07:03.268205609 +0000 UTC m=+51.385685052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/22411ad6-db7e-412f-9088-4da16631a9cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhnn4" (UID: "22411ad6-db7e-412f-9088-4da16631a9cf") : configmap "kube-rbac-proxy" not found Sep 17 23:07:01.268534 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:01.268301 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 23:07:01.268534 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:01.268360 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert podName:8b5569f3-268e-4bd8-b7dc-6f5f0102f08c nodeName:}" failed. No retries permitted until 2026-09-17 23:07:09.268345692 +0000 UTC m=+57.385825127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-dp5fl" (UID: "8b5569f3-268e-4bd8-b7dc-6f5f0102f08c") : secret "networking-console-plugin-cert" not found Sep 17 23:07:01.388207 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.388162 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-glczx"] Sep 17 23:07:01.403643 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.403601 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-glczx"] Sep 17 23:07:01.403806 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.403743 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" Sep 17 23:07:01.407844 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.407770 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 17 23:07:01.408028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.407856 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 17 23:07:01.408827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.408708 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 17 23:07:01.408827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.408763 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 17 23:07:01.408827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.408790 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8wgfl\"" Sep 17 23:07:01.442150 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.442116 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:01.470319 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.470289 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1dcfe94f-3d8b-4902-a084-b2493aacb510-signing-key\") pod \"service-ca-55f6fc7b86-glczx\" (UID: \"1dcfe94f-3d8b-4902-a084-b2493aacb510\") " pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" Sep 17 23:07:01.470511 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.470345 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1dcfe94f-3d8b-4902-a084-b2493aacb510-signing-cabundle\") pod \"service-ca-55f6fc7b86-glczx\" (UID: \"1dcfe94f-3d8b-4902-a084-b2493aacb510\") " pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" Sep 17 23:07:01.470511 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.470374 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4tt2\" (UniqueName: \"kubernetes.io/projected/1dcfe94f-3d8b-4902-a084-b2493aacb510-kube-api-access-l4tt2\") pod \"service-ca-55f6fc7b86-glczx\" (UID: \"1dcfe94f-3d8b-4902-a084-b2493aacb510\") " pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" Sep 17 23:07:01.571842 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.571752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4tt2\" (UniqueName: \"kubernetes.io/projected/1dcfe94f-3d8b-4902-a084-b2493aacb510-kube-api-access-l4tt2\") pod \"service-ca-55f6fc7b86-glczx\" (UID: \"1dcfe94f-3d8b-4902-a084-b2493aacb510\") " pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" Sep 17 23:07:01.572017 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.571912 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1dcfe94f-3d8b-4902-a084-b2493aacb510-signing-key\") pod \"service-ca-55f6fc7b86-glczx\" (UID: \"1dcfe94f-3d8b-4902-a084-b2493aacb510\") " pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" Sep 17 23:07:01.572086 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.572071 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1dcfe94f-3d8b-4902-a084-b2493aacb510-signing-cabundle\") pod \"service-ca-55f6fc7b86-glczx\" (UID: \"1dcfe94f-3d8b-4902-a084-b2493aacb510\") " pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" Sep 17 23:07:01.572950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.572925 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1dcfe94f-3d8b-4902-a084-b2493aacb510-signing-cabundle\") pod \"service-ca-55f6fc7b86-glczx\" (UID: \"1dcfe94f-3d8b-4902-a084-b2493aacb510\") " pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" Sep 17 23:07:01.574600 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.574572 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1dcfe94f-3d8b-4902-a084-b2493aacb510-signing-key\") pod \"service-ca-55f6fc7b86-glczx\" (UID: \"1dcfe94f-3d8b-4902-a084-b2493aacb510\") " pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" Sep 17 23:07:01.580670 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.580646 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4tt2\" (UniqueName: \"kubernetes.io/projected/1dcfe94f-3d8b-4902-a084-b2493aacb510-kube-api-access-l4tt2\") pod \"service-ca-55f6fc7b86-glczx\" (UID: \"1dcfe94f-3d8b-4902-a084-b2493aacb510\") " pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" Sep 17 23:07:01.715257 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.715204 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" Sep 17 23:07:01.752330 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:01.752293 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l5s9l_064ea6f5-bd51-42ec-825e-25f7ea4a6723/node-ca/0.log" Sep 17 23:07:02.079603 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:02.079572 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-glczx"] Sep 17 23:07:02.087425 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:02.087397 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcfe94f_3d8b_4902_a084_b2493aacb510.slice/crio-2a2d2f191926c24ddcbb0231ac62f5aabfd3a3ea2e2cf3a0f0fc42fca57f66cc WatchSource:0}: Error finding container 2a2d2f191926c24ddcbb0231ac62f5aabfd3a3ea2e2cf3a0f0fc42fca57f66cc: Status 404 returned error can't find the container with id 2a2d2f191926c24ddcbb0231ac62f5aabfd3a3ea2e2cf3a0f0fc42fca57f66cc Sep 17 23:07:02.318089 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:02.318062 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:02.357235 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:02.357214 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6xlrg_d16d7679-5919-42cb-b4c7-379fadd35925/serve-healthcheck-canary/0.log" Sep 17 23:07:02.673791 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:02.673750 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r8t9z" event={"ID":"252f319b-a647-4f72-915c-2a4df27d83fe","Type":"ContainerStarted","Data":"2a152081476ad7c212cb1328c23df92a31444390342612bcff0840de9b36c03a"} Sep 17 23:07:02.674024 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:02.673814 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r8t9z" event={"ID":"252f319b-a647-4f72-915c-2a4df27d83fe","Type":"ContainerStarted","Data":"f35e72d2ed2eccb0ede451d8f0e2a3dad94cc2340116710a1ed3eee9d18447a3"} Sep 17 23:07:02.675285 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:02.675261 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" event={"ID":"1dcfe94f-3d8b-4902-a084-b2493aacb510","Type":"ContainerStarted","Data":"54be52db753864bad33ed962cbefc8bbcc4fa166cce7b5075fff32b5645600d9"} Sep 17 23:07:02.675400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:02.675298 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" event={"ID":"1dcfe94f-3d8b-4902-a084-b2493aacb510","Type":"ContainerStarted","Data":"2a2d2f191926c24ddcbb0231ac62f5aabfd3a3ea2e2cf3a0f0fc42fca57f66cc"} Sep 17 23:07:02.692331 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:02.692285 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r8t9z" podStartSLOduration=2.159170553 podStartE2EDuration="3.692268185s" totalImagesPullingTime="1.533097632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:02.690950217 +0000 UTC m=+50.808429689" watchObservedRunningTime="2026-09-17 23:07:02.692268185 +0000 UTC m=+50.809747670" Sep 17 23:07:02.706439 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:02.706385 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-glczx" podStartSLOduration=1.7063692019999999 podStartE2EDuration="1.706369202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:02.70603311 +0000 UTC m=+50.823512591" watchObservedRunningTime="2026-09-17 23:07:02.706369202 +0000 UTC m=+50.823848652" Sep 17 23:07:03.085199 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:03.085156 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:07:03.085199 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:03.085200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:07:03.085747 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:03.085337 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 23:07:03.085747 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:03.085340 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:07:03.085747 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:03.085427 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls podName:5b7ac966-8594-4a9f-aa3f-85f12087b2a3 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:19.085406314 +0000 UTC m=+67.202885763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls") pod "dns-default-2fdkb" (UID: "5b7ac966-8594-4a9f-aa3f-85f12087b2a3") : secret "dns-default-metrics-tls" not found Sep 17 23:07:03.085747 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:03.085348 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f64f764df-gtws9: secret "image-registry-tls" not found Sep 17 23:07:03.085747 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:03.085479 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls podName:ce53708a-abfa-4b74-b949-2456ce5cd297 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:19.085465443 +0000 UTC m=+67.202944873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls") pod "image-registry-7f64f764df-gtws9" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297") : secret "image-registry-tls" not found Sep 17 23:07:03.113037 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:03.113002 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:03.287776 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:03.287728 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22411ad6-db7e-412f-9088-4da16631a9cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:07:03.288001 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:03.287958 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:07:03.288098 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:03.288084 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 23:07:03.288156 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:03.288139 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls podName:22411ad6-db7e-412f-9088-4da16631a9cf nodeName:}" failed. No retries permitted until 2026-09-17 23:07:07.288125809 +0000 UTC m=+55.405605268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhnn4" (UID: "22411ad6-db7e-412f-9088-4da16631a9cf") : secret "insights-runtime-extractor-tls" not found Sep 17 23:07:03.288307 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:03.288288 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22411ad6-db7e-412f-9088-4da16631a9cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:07:03.668728 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:03.668696 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:03.782702 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:03.782666 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:03.868793 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:03.868754 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:03.969599 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:03.969566 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:04.069650 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:04.069613 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:04.972406 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:04.972363 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:05.003488 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:05.003455 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:07:05.003622 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:05.003518 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-lrvzv\" (UID: \"c27493ad-ac0b-46cb-a7bd-92da0fb886bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:07:05.003670 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:05.003629 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 23:07:05.003721 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:05.003708 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls podName:b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:21.003686475 +0000 UTC m=+69.121165910 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-r2vwj" (UID: "b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5") : secret "cluster-monitoring-operator-tls" not found Sep 17 23:07:05.006160 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:05.006137 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c27493ad-ac0b-46cb-a7bd-92da0fb886bc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-lrvzv\" (UID: \"c27493ad-ac0b-46cb-a7bd-92da0fb886bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:07:05.040336 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:05.040305 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" Sep 17 23:07:05.079262 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:05.079230 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:05.179341 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:05.179310 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" limit=1024 Sep 17 23:07:05.205324 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:05.205284 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:07:05.205484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:05.205341 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:07:05.205484 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:05.205476 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle podName:0938e02b-9bf2-4f98-92b4-ee86260aef8f nodeName:}" failed. No retries permitted until 2026-09-17 23:07:21.205454387 +0000 UTC m=+69.322933836 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle") pod "router-default-5cd8bf6458-jhsss" (UID: "0938e02b-9bf2-4f98-92b4-ee86260aef8f") : configmap references non-existent config key: service-ca.crt Sep 17 23:07:05.205636 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:05.205529 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 23:07:05.205636 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:05.205596 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs podName:0938e02b-9bf2-4f98-92b4-ee86260aef8f nodeName:}" failed. No retries permitted until 2026-09-17 23:07:21.205579346 +0000 UTC m=+69.323058779 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs") pod "router-default-5cd8bf6458-jhsss" (UID: "0938e02b-9bf2-4f98-92b4-ee86260aef8f") : secret "router-metrics-certs-default" not found Sep 17 23:07:05.361479 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:05.361406 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv"] Sep 17 23:07:05.688457 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:05.688372 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" event={"ID":"c27493ad-ac0b-46cb-a7bd-92da0fb886bc","Type":"ContainerStarted","Data":"ed891294b6449a8a26c8c9d894313f8df2919c43c634a9ba218b45b5267808d4"} Sep 17 23:07:07.323876 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:07.323784 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:07:07.324318 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:07.323928 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 23:07:07.324318 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:07.323988 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls podName:22411ad6-db7e-412f-9088-4da16631a9cf nodeName:}" failed. No retries permitted until 2026-09-17 23:07:15.323971077 +0000 UTC m=+63.441450506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhnn4" (UID: "22411ad6-db7e-412f-9088-4da16631a9cf") : secret "insights-runtime-extractor-tls" not found Sep 17 23:07:07.697630 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:07.697594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" event={"ID":"c27493ad-ac0b-46cb-a7bd-92da0fb886bc","Type":"ContainerStarted","Data":"ea5035978f122efc487c0b3fb671e4932f0f1790e0d4478d422721321ec9d60d"} Sep 17 23:07:07.697802 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:07.697677 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" event={"ID":"c27493ad-ac0b-46cb-a7bd-92da0fb886bc","Type":"ContainerStarted","Data":"46f1e5704b6596ee7c4a377bdf0403fa1a9da1089e0f00b0f328093e02c06747"} Sep 17 23:07:07.716999 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:07.716949 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-lrvzv" podStartSLOduration=17.211542316 podStartE2EDuration="18.716932926s" totalImagesPullingTime="1.50539061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:07.716498848 +0000 UTC m=+55.833978306" watchObservedRunningTime="2026-09-17 23:07:07.716932926 +0000 UTC m=+55.834412386" Sep 17 23:07:09.346614 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:09.346579 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dp5fl\" (UID: \"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:07:09.347128 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:09.346758 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 23:07:09.347128 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:07:09.346835 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert podName:8b5569f3-268e-4bd8-b7dc-6f5f0102f08c nodeName:}" failed. No retries permitted until 2026-09-17 23:07:25.346814377 +0000 UTC m=+73.464293812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-dp5fl" (UID: "8b5569f3-268e-4bd8-b7dc-6f5f0102f08c") : secret "networking-console-plugin-cert" not found Sep 17 23:07:09.451375 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:09.451331 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:07:09.451577 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:09.451410 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:07:09.451883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:09.451868 2559 scope.go:117] "RemoveContainer" containerID="22a63b413aad3867ac243027ea85ee65433c94006150533fb5d9986cf0022041" Sep 17 23:07:09.706171 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:09.706142 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:07:09.706333 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:09.706227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" event={"ID":"c69cad18-c083-405a-aed3-91740cc79fd4","Type":"ContainerStarted","Data":"d174bd90abc3f9a044f4b27e551beb5a66d68a8120fb332d9fa3fc6ee5973e64"} Sep 17 23:07:09.706594 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:09.706564 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:07:09.726891 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:09.726823 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" podStartSLOduration=14.387585639 podStartE2EDuration="20.726801192s" totalImagesPullingTime="6.339215553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:09.72528098 +0000 UTC m=+57.842760432" watchObservedRunningTime="2026-09-17 23:07:09.726801192 +0000 UTC m=+57.844280644" Sep 17 23:07:09.997263 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:09.997188 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-kfrv8" Sep 17 23:07:12.552109 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:12.552077 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljnfz" Sep 17 23:07:15.400415 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:15.400375 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:07:15.402887 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:15.402860 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22411ad6-db7e-412f-9088-4da16631a9cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhnn4\" (UID: \"22411ad6-db7e-412f-9088-4da16631a9cf\") " pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:07:15.530766 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:15.530734 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ktswr\"" Sep 17 23:07:15.538719 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:15.538695 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhnn4" Sep 17 23:07:15.810265 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:15.810173 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vhnn4"] Sep 17 23:07:15.813886 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:15.813856 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22411ad6_db7e_412f_9088_4da16631a9cf.slice/crio-e9115187ac3d59a3d28412ad378dea5f683e186d41b4a6a15cd7965492ef6f5b WatchSource:0}: Error finding container e9115187ac3d59a3d28412ad378dea5f683e186d41b4a6a15cd7965492ef6f5b: Status 404 returned error can't find the container with id e9115187ac3d59a3d28412ad378dea5f683e186d41b4a6a15cd7965492ef6f5b Sep 17 23:07:16.731052 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:16.731015 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhnn4" event={"ID":"22411ad6-db7e-412f-9088-4da16631a9cf","Type":"ContainerStarted","Data":"1a2ec29585f4fedfee358aab446996e278fe726090e1688d1a3c20b30ba88813"} Sep 17 23:07:16.731424 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:16.731076 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhnn4" event={"ID":"22411ad6-db7e-412f-9088-4da16631a9cf","Type":"ContainerStarted","Data":"a4377d26b510c2fc421be46db6a1007adc090c5bc019aba70b1ef927487f3506"} Sep 17 23:07:16.731424 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:16.731096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhnn4" event={"ID":"22411ad6-db7e-412f-9088-4da16631a9cf","Type":"ContainerStarted","Data":"e9115187ac3d59a3d28412ad378dea5f683e186d41b4a6a15cd7965492ef6f5b"} Sep 17 23:07:18.743996 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:18.743959 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhnn4" event={"ID":"22411ad6-db7e-412f-9088-4da16631a9cf","Type":"ContainerStarted","Data":"f7ec4cc2c051c013797a7ff0621bcda8e592b016528399b8feeb8b14b435f732"} Sep 17 23:07:18.766825 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:18.766774 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vhnn4" podStartSLOduration=17.831774039 podStartE2EDuration="19.766759959s" totalImagesPullingTime="1.93498592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:18.765689132 +0000 UTC m=+66.883168598" watchObservedRunningTime="2026-09-17 23:07:18.766759959 +0000 UTC m=+66.884239410" Sep 17 23:07:18.930470 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:18.930419 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlff5\" (UniqueName: \"kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5\") pod \"network-check-target-sq8mp\" (UID: \"13108b6b-4ed6-4ca0-893a-fd49f1ae5b78\") " pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:07:18.930659 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:18.930525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:07:18.930659 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:18.930562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:07:18.933135 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:18.933103 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlff5\" (UniqueName: \"kubernetes.io/projected/13108b6b-4ed6-4ca0-893a-fd49f1ae5b78-kube-api-access-xlff5\") pod \"network-check-target-sq8mp\" (UID: \"13108b6b-4ed6-4ca0-893a-fd49f1ae5b78\") " pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:07:18.933135 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:18.933131 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 23:07:18.933324 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:18.933134 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 23:07:18.943152 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:18.943128 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4512e542-b8f7-4833-b2a5-9812a90f43c6-original-pull-secret\") pod \"global-pull-secret-syncer-92l5f\" (UID: \"4512e542-b8f7-4833-b2a5-9812a90f43c6\") " pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:07:18.943286 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:18.943129 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1ba703-ed87-422b-a256-bc96859795af-metrics-certs\") pod \"network-metrics-daemon-vk29c\" (UID: \"ea1ba703-ed87-422b-a256-bc96859795af\") " pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:07:19.092369 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.092285 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kxqn4\"" Sep 17 23:07:19.100410 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.100380 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:07:19.132475 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.132433 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:07:19.132475 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.132476 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:07:19.134789 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.134762 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b7ac966-8594-4a9f-aa3f-85f12087b2a3-metrics-tls\") pod \"dns-default-2fdkb\" (UID: \"5b7ac966-8594-4a9f-aa3f-85f12087b2a3\") " pod="openshift-dns/dns-default-2fdkb" Sep 17 23:07:19.134930 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.134885 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls\") pod \"image-registry-7f64f764df-gtws9\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:07:19.194495 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.194448 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l5f" Sep 17 23:07:19.203498 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.203470 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t46xp\"" Sep 17 23:07:19.211561 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.211536 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vk29c" Sep 17 23:07:19.292759 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.292730 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cm7l8\"" Sep 17 23:07:19.300754 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.300728 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:07:19.308803 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.308785 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vvv69\"" Sep 17 23:07:19.316816 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.316795 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2fdkb" Sep 17 23:07:19.374607 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.374373 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sq8mp"] Sep 17 23:07:19.378269 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:19.378225 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13108b6b_4ed6_4ca0_893a_fd49f1ae5b78.slice/crio-cf018181b3372e232e8124f6689e89c6b5906db5f1c834d727091a45cca1c056 WatchSource:0}: Error finding container cf018181b3372e232e8124f6689e89c6b5906db5f1c834d727091a45cca1c056: Status 404 returned error can't find the container with id cf018181b3372e232e8124f6689e89c6b5906db5f1c834d727091a45cca1c056 Sep 17 23:07:19.463331 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.463302 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-92l5f"] Sep 17 23:07:19.466496 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:19.466473 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4512e542_b8f7_4833_b2a5_9812a90f43c6.slice/crio-dee9f57593c042a6cf92f5a0d5ba13c3799abe8e37a2d5248abce7460af6b464 WatchSource:0}: Error finding container dee9f57593c042a6cf92f5a0d5ba13c3799abe8e37a2d5248abce7460af6b464: Status 404 returned error can't find the container with id dee9f57593c042a6cf92f5a0d5ba13c3799abe8e37a2d5248abce7460af6b464 Sep 17 23:07:19.487078 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.487048 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vk29c"] Sep 17 23:07:19.490226 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:19.490198 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1ba703_ed87_422b_a256_bc96859795af.slice/crio-894f5c35ab057e4ca0470167e1b42e82ce805f60c5f5e9c136a3a62aed15b3d4 WatchSource:0}: Error finding container 894f5c35ab057e4ca0470167e1b42e82ce805f60c5f5e9c136a3a62aed15b3d4: Status 404 returned error can't find the container with id 894f5c35ab057e4ca0470167e1b42e82ce805f60c5f5e9c136a3a62aed15b3d4 Sep 17 23:07:19.572919 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.572716 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f64f764df-gtws9"] Sep 17 23:07:19.575414 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:19.575389 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce53708a_abfa_4b74_b949_2456ce5cd297.slice/crio-fc73ef36871523f7b1e7a142a16335787e68022c0bf1a07ed38f5be6fa58d518 WatchSource:0}: Error finding container fc73ef36871523f7b1e7a142a16335787e68022c0bf1a07ed38f5be6fa58d518: Status 404 returned error can't find the container with id fc73ef36871523f7b1e7a142a16335787e68022c0bf1a07ed38f5be6fa58d518 Sep 17 23:07:19.595496 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.595469 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2fdkb"] Sep 17 23:07:19.603069 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:19.603032 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7ac966_8594_4a9f_aa3f_85f12087b2a3.slice/crio-56a609d20ab8191a3ae3f4a01e63508e502366168888249230f60bf220d42a57 WatchSource:0}: Error finding container 56a609d20ab8191a3ae3f4a01e63508e502366168888249230f60bf220d42a57: Status 404 returned error can't find the container with id 56a609d20ab8191a3ae3f4a01e63508e502366168888249230f60bf220d42a57 Sep 17 23:07:19.749422 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.749382 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" event={"ID":"ce53708a-abfa-4b74-b949-2456ce5cd297","Type":"ContainerStarted","Data":"bba548b94e8251e0de609cd20b8479f79c631c301e3efa879ce19cb6a4c9e7fa"} Sep 17 23:07:19.749874 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.749437 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" event={"ID":"ce53708a-abfa-4b74-b949-2456ce5cd297","Type":"ContainerStarted","Data":"fc73ef36871523f7b1e7a142a16335787e68022c0bf1a07ed38f5be6fa58d518"} Sep 17 23:07:19.749874 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.749486 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:07:19.750952 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.750924 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vk29c" event={"ID":"ea1ba703-ed87-422b-a256-bc96859795af","Type":"ContainerStarted","Data":"894f5c35ab057e4ca0470167e1b42e82ce805f60c5f5e9c136a3a62aed15b3d4"} Sep 17 23:07:19.752212 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.752184 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-92l5f" event={"ID":"4512e542-b8f7-4833-b2a5-9812a90f43c6","Type":"ContainerStarted","Data":"dee9f57593c042a6cf92f5a0d5ba13c3799abe8e37a2d5248abce7460af6b464"} Sep 17 23:07:19.753824 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.753791 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sq8mp" event={"ID":"13108b6b-4ed6-4ca0-893a-fd49f1ae5b78","Type":"ContainerStarted","Data":"23612b6324ec8c5cf539a48921b31ff93352dd8672a700ed2137b33114c4dcee"} Sep 17 23:07:19.753952 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.753828 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sq8mp" event={"ID":"13108b6b-4ed6-4ca0-893a-fd49f1ae5b78","Type":"ContainerStarted","Data":"cf018181b3372e232e8124f6689e89c6b5906db5f1c834d727091a45cca1c056"} Sep 17 23:07:19.753952 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.753848 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:07:19.754988 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.754966 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2fdkb" event={"ID":"5b7ac966-8594-4a9f-aa3f-85f12087b2a3","Type":"ContainerStarted","Data":"56a609d20ab8191a3ae3f4a01e63508e502366168888249230f60bf220d42a57"} Sep 17 23:07:19.771143 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.771091 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" podStartSLOduration=65.771075901 podStartE2EDuration="1m5.771075901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:19.770411639 +0000 UTC m=+67.887891116" watchObservedRunningTime="2026-09-17 23:07:19.771075901 +0000 UTC m=+67.888555351" Sep 17 23:07:19.787448 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:19.787397 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sq8mp" podStartSLOduration=65.787379916 podStartE2EDuration="1m5.787379916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:19.786658343 +0000 UTC m=+67.904137816" watchObservedRunningTime="2026-09-17 23:07:19.787379916 +0000 UTC m=+67.904859366" Sep 17 23:07:20.762226 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:20.762134 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vk29c" event={"ID":"ea1ba703-ed87-422b-a256-bc96859795af","Type":"ContainerStarted","Data":"e33ae121dc26f188f5574477148ad4099bb5098c8789d549c5796837053e1296"} Sep 17 23:07:20.762226 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:20.762206 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vk29c" event={"ID":"ea1ba703-ed87-422b-a256-bc96859795af","Type":"ContainerStarted","Data":"62213c5754597efcea5ff5aa317ec6fe1c81125568548383347e21a05e3e99c1"} Sep 17 23:07:20.779384 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:20.779273 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vk29c" podStartSLOduration=65.780787889 podStartE2EDuration="1m6.779250869s" totalImagesPullingTime="998.46298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:20.778371217 +0000 UTC m=+68.895850713" watchObservedRunningTime="2026-09-17 23:07:20.779250869 +0000 UTC m=+68.896730322" Sep 17 23:07:21.052369 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.052274 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:07:21.055398 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.055362 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-r2vwj\" (UID: \"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:07:21.109060 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.109012 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz"] Sep 17 23:07:21.112248 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.112222 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc"] Sep 17 23:07:21.112881 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.112854 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:21.115741 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.115335 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf"] Sep 17 23:07:21.115741 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.115468 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc" Sep 17 23:07:21.116106 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.116085 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 17 23:07:21.116181 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.116150 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 17 23:07:21.116262 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.116091 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 17 23:07:21.116376 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.116361 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 17 23:07:21.117872 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.117852 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.118116 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.118012 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-42xnk\"" Sep 17 23:07:21.118116 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.118043 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 17 23:07:21.120487 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.120465 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 17 23:07:21.120610 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.120527 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 17 23:07:21.120610 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.120531 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 17 23:07:21.120722 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.120474 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 17 23:07:21.127786 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.127751 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz"] Sep 17 23:07:21.131858 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.131084 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc"] Sep 17 23:07:21.132789 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.132768 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf"] Sep 17 23:07:21.227362 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.227326 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-kzt9c"] Sep 17 23:07:21.230122 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.230097 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-kzt9c" Sep 17 23:07:21.235673 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.235642 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 23:07:21.238104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.238082 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 23:07:21.238233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.238191 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-64ksw\"" Sep 17 23:07:21.238314 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.238299 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8njmm\"" Sep 17 23:07:21.241229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.241172 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" Sep 17 23:07:21.246747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.246717 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-kzt9c"] Sep 17 23:07:21.254744 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.254717 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:07:21.254863 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.254749 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e950bcd4-51a3-45da-b5d1-f1e22867a554-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.254863 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.254772 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e950bcd4-51a3-45da-b5d1-f1e22867a554-hub\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.255015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.254882 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e950bcd4-51a3-45da-b5d1-f1e22867a554-ca\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.255015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.254935 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e950bcd4-51a3-45da-b5d1-f1e22867a554-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.255015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.254964 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e950bcd4-51a3-45da-b5d1-f1e22867a554-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.255015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.254990 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvr9\" (UniqueName: \"kubernetes.io/projected/838468ce-22e4-47fd-a310-17131527ca89-kube-api-access-whvr9\") pod \"klusterlet-addon-workmgr-59677dcbd5-9szhz\" (UID: \"838468ce-22e4-47fd-a310-17131527ca89\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:21.255211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.255028 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:07:21.255211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.255048 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwbg\" (UniqueName: \"kubernetes.io/projected/033e276e-f5de-4098-b9ce-73abfcee38a3-kube-api-access-nwwbg\") pod \"managed-serviceaccount-addon-agent-5cf8885b78-rc5xc\" (UID: \"033e276e-f5de-4098-b9ce-73abfcee38a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc" Sep 17 23:07:21.255211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.255105 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5l8\" (UniqueName: \"kubernetes.io/projected/e950bcd4-51a3-45da-b5d1-f1e22867a554-kube-api-access-qn5l8\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.255211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.255150 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/033e276e-f5de-4098-b9ce-73abfcee38a3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cf8885b78-rc5xc\" (UID: \"033e276e-f5de-4098-b9ce-73abfcee38a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc" Sep 17 23:07:21.255211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.255175 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/838468ce-22e4-47fd-a310-17131527ca89-tmp\") pod \"klusterlet-addon-workmgr-59677dcbd5-9szhz\" (UID: \"838468ce-22e4-47fd-a310-17131527ca89\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:21.255444 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.255211 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/838468ce-22e4-47fd-a310-17131527ca89-klusterlet-config\") pod \"klusterlet-addon-workmgr-59677dcbd5-9szhz\" (UID: \"838468ce-22e4-47fd-a310-17131527ca89\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:21.255664 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.255615 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0938e02b-9bf2-4f98-92b4-ee86260aef8f-service-ca-bundle\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:07:21.257860 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.257838 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0938e02b-9bf2-4f98-92b4-ee86260aef8f-metrics-certs\") pod \"router-default-5cd8bf6458-jhsss\" (UID: \"0938e02b-9bf2-4f98-92b4-ee86260aef8f\") " pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:07:21.356300 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.356174 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e950bcd4-51a3-45da-b5d1-f1e22867a554-ca\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.356300 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.356239 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e950bcd4-51a3-45da-b5d1-f1e22867a554-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.356300 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.356264 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e950bcd4-51a3-45da-b5d1-f1e22867a554-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.356606 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.356317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvr9\" (UniqueName: \"kubernetes.io/projected/838468ce-22e4-47fd-a310-17131527ca89-kube-api-access-whvr9\") pod \"klusterlet-addon-workmgr-59677dcbd5-9szhz\" (UID: \"838468ce-22e4-47fd-a310-17131527ca89\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:21.356606 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.356349 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwbg\" (UniqueName: \"kubernetes.io/projected/033e276e-f5de-4098-b9ce-73abfcee38a3-kube-api-access-nwwbg\") pod \"managed-serviceaccount-addon-agent-5cf8885b78-rc5xc\" (UID: \"033e276e-f5de-4098-b9ce-73abfcee38a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc" Sep 17 23:07:21.356606 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.356410 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5l8\" (UniqueName: \"kubernetes.io/projected/e950bcd4-51a3-45da-b5d1-f1e22867a554-kube-api-access-qn5l8\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.356606 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.356466 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/033e276e-f5de-4098-b9ce-73abfcee38a3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cf8885b78-rc5xc\" (UID: \"033e276e-f5de-4098-b9ce-73abfcee38a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc" Sep 17 23:07:21.356606 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.356505 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/838468ce-22e4-47fd-a310-17131527ca89-tmp\") pod \"klusterlet-addon-workmgr-59677dcbd5-9szhz\" (UID: \"838468ce-22e4-47fd-a310-17131527ca89\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:21.356606 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.356531 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8b4\" (UniqueName: \"kubernetes.io/projected/46c568d6-e39e-4f20-89a9-376e4b667e20-kube-api-access-ph8b4\") pod \"downloads-6594f4c75-kzt9c\" (UID: \"46c568d6-e39e-4f20-89a9-376e4b667e20\") " pod="openshift-console/downloads-6594f4c75-kzt9c" Sep 17 23:07:21.356606 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.356577 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/838468ce-22e4-47fd-a310-17131527ca89-klusterlet-config\") pod \"klusterlet-addon-workmgr-59677dcbd5-9szhz\" (UID: \"838468ce-22e4-47fd-a310-17131527ca89\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:21.356952 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.356608 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e950bcd4-51a3-45da-b5d1-f1e22867a554-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.356952 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.356645 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e950bcd4-51a3-45da-b5d1-f1e22867a554-hub\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.357494 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.357459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e950bcd4-51a3-45da-b5d1-f1e22867a554-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.357611 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.357584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/838468ce-22e4-47fd-a310-17131527ca89-tmp\") pod \"klusterlet-addon-workmgr-59677dcbd5-9szhz\" (UID: \"838468ce-22e4-47fd-a310-17131527ca89\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:21.359404 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.359374 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e950bcd4-51a3-45da-b5d1-f1e22867a554-ca\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.359807 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.359783 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e950bcd4-51a3-45da-b5d1-f1e22867a554-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.361174 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.361149 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e950bcd4-51a3-45da-b5d1-f1e22867a554-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.361351 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.361301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e950bcd4-51a3-45da-b5d1-f1e22867a554-hub\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.361507 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.361487 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/033e276e-f5de-4098-b9ce-73abfcee38a3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cf8885b78-rc5xc\" (UID: \"033e276e-f5de-4098-b9ce-73abfcee38a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc" Sep 17 23:07:21.362456 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.362430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/838468ce-22e4-47fd-a310-17131527ca89-klusterlet-config\") pod \"klusterlet-addon-workmgr-59677dcbd5-9szhz\" (UID: \"838468ce-22e4-47fd-a310-17131527ca89\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:21.371288 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.371233 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5l8\" (UniqueName: \"kubernetes.io/projected/e950bcd4-51a3-45da-b5d1-f1e22867a554-kube-api-access-qn5l8\") pod \"cluster-proxy-proxy-agent-69cb9cb6f5-h7svf\" (UID: \"e950bcd4-51a3-45da-b5d1-f1e22867a554\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.371288 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.371260 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvr9\" (UniqueName: \"kubernetes.io/projected/838468ce-22e4-47fd-a310-17131527ca89-kube-api-access-whvr9\") pod \"klusterlet-addon-workmgr-59677dcbd5-9szhz\" (UID: \"838468ce-22e4-47fd-a310-17131527ca89\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:21.373085 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.373011 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwbg\" (UniqueName: \"kubernetes.io/projected/033e276e-f5de-4098-b9ce-73abfcee38a3-kube-api-access-nwwbg\") pod \"managed-serviceaccount-addon-agent-5cf8885b78-rc5xc\" (UID: \"033e276e-f5de-4098-b9ce-73abfcee38a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc" Sep 17 23:07:21.419886 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.419839 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vv779\"" Sep 17 23:07:21.427817 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.427717 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:07:21.430081 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.430047 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:21.457533 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.457491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8b4\" (UniqueName: \"kubernetes.io/projected/46c568d6-e39e-4f20-89a9-376e4b667e20-kube-api-access-ph8b4\") pod \"downloads-6594f4c75-kzt9c\" (UID: \"46c568d6-e39e-4f20-89a9-376e4b667e20\") " pod="openshift-console/downloads-6594f4c75-kzt9c" Sep 17 23:07:21.460470 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.460446 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc" Sep 17 23:07:21.467824 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.467786 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" Sep 17 23:07:21.473021 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.472996 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8b4\" (UniqueName: \"kubernetes.io/projected/46c568d6-e39e-4f20-89a9-376e4b667e20-kube-api-access-ph8b4\") pod \"downloads-6594f4c75-kzt9c\" (UID: \"46c568d6-e39e-4f20-89a9-376e4b667e20\") " pod="openshift-console/downloads-6594f4c75-kzt9c" Sep 17 23:07:21.552596 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.552490 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-kzt9c" Sep 17 23:07:21.771422 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.771357 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2fdkb" event={"ID":"5b7ac966-8594-4a9f-aa3f-85f12087b2a3","Type":"ContainerStarted","Data":"b44caca2104bef47250bd2159018c30b154ca5c8b443e5b8f61a57d78d411cb0"} Sep 17 23:07:21.878425 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.878361 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cd8bf6458-jhsss"] Sep 17 23:07:21.881840 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:21.881810 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0938e02b_9bf2_4f98_92b4_ee86260aef8f.slice/crio-30e638bbdb0e1c9c06775c40f54a0d2f5f6a2db7e1e3125941eeed9d2fb62081 WatchSource:0}: Error finding container 30e638bbdb0e1c9c06775c40f54a0d2f5f6a2db7e1e3125941eeed9d2fb62081: Status 404 returned error can't find the container with id 30e638bbdb0e1c9c06775c40f54a0d2f5f6a2db7e1e3125941eeed9d2fb62081 Sep 17 23:07:21.882435 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.882415 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj"] Sep 17 23:07:21.888341 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.887846 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz"] Sep 17 23:07:21.889328 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:21.888556 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a856fa_a9da_41e6_8a80_da9ddc2d5cb5.slice/crio-c9f8cabe244ca3865b4b40f1db0e7b91ca3cf5d1f4b159d58720f3b27dfcdd8d WatchSource:0}: Error finding container c9f8cabe244ca3865b4b40f1db0e7b91ca3cf5d1f4b159d58720f3b27dfcdd8d: Status 404 returned error can't find the container with id c9f8cabe244ca3865b4b40f1db0e7b91ca3cf5d1f4b159d58720f3b27dfcdd8d Sep 17 23:07:21.889472 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.889372 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc"] Sep 17 23:07:21.891369 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:21.890595 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod033e276e_f5de_4098_b9ce_73abfcee38a3.slice/crio-22f73805e9d6b631b81555ddaff21b15ee1fb183875c1f13e03473be12f5b97f WatchSource:0}: Error finding container 22f73805e9d6b631b81555ddaff21b15ee1fb183875c1f13e03473be12f5b97f: Status 404 returned error can't find the container with id 22f73805e9d6b631b81555ddaff21b15ee1fb183875c1f13e03473be12f5b97f Sep 17 23:07:21.891369 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.891030 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf"] Sep 17 23:07:21.891540 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:21.891460 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838468ce_22e4_47fd_a310_17131527ca89.slice/crio-46aef04b2f96a06e088da12f5759e043508404c9c513c5a3da5b44ef592425a1 WatchSource:0}: Error finding container 46aef04b2f96a06e088da12f5759e043508404c9c513c5a3da5b44ef592425a1: Status 404 returned error can't find the container with id 46aef04b2f96a06e088da12f5759e043508404c9c513c5a3da5b44ef592425a1 Sep 17 23:07:21.920167 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:21.920132 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-kzt9c"] Sep 17 23:07:21.923093 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:21.923065 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c568d6_e39e_4f20_89a9_376e4b667e20.slice/crio-7acc3d1f44071277056980debbab78e122a700cf9769e75b493010cbc421b949 WatchSource:0}: Error finding container 7acc3d1f44071277056980debbab78e122a700cf9769e75b493010cbc421b949: Status 404 returned error can't find the container with id 7acc3d1f44071277056980debbab78e122a700cf9769e75b493010cbc421b949 Sep 17 23:07:22.777412 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:22.777371 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" event={"ID":"838468ce-22e4-47fd-a310-17131527ca89","Type":"ContainerStarted","Data":"46aef04b2f96a06e088da12f5759e043508404c9c513c5a3da5b44ef592425a1"} Sep 17 23:07:22.779428 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:22.779393 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cd8bf6458-jhsss" event={"ID":"0938e02b-9bf2-4f98-92b4-ee86260aef8f","Type":"ContainerStarted","Data":"cd51528acd7889a73e7008a55327325723d7aafc36befa7a043a8964bed83ab1"} Sep 17 23:07:22.779575 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:22.779450 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cd8bf6458-jhsss" event={"ID":"0938e02b-9bf2-4f98-92b4-ee86260aef8f","Type":"ContainerStarted","Data":"30e638bbdb0e1c9c06775c40f54a0d2f5f6a2db7e1e3125941eeed9d2fb62081"} Sep 17 23:07:22.780970 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:22.780893 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" event={"ID":"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5","Type":"ContainerStarted","Data":"c9f8cabe244ca3865b4b40f1db0e7b91ca3cf5d1f4b159d58720f3b27dfcdd8d"} Sep 17 23:07:22.782327 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:22.782288 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" event={"ID":"e950bcd4-51a3-45da-b5d1-f1e22867a554","Type":"ContainerStarted","Data":"670d8c22c8a6c3c09199227c9c3be33165ac6e9170c6af360bf11d6743d71e73"} Sep 17 23:07:22.783834 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:22.783805 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc" event={"ID":"033e276e-f5de-4098-b9ce-73abfcee38a3","Type":"ContainerStarted","Data":"22f73805e9d6b631b81555ddaff21b15ee1fb183875c1f13e03473be12f5b97f"} Sep 17 23:07:22.786100 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:22.786075 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2fdkb" event={"ID":"5b7ac966-8594-4a9f-aa3f-85f12087b2a3","Type":"ContainerStarted","Data":"4b7c0aad3771c987b413173545f145fffc718b64516b45b7d3bf41b268673b4d"} Sep 17 23:07:22.786240 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:22.786217 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2fdkb" Sep 17 23:07:22.787488 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:22.787455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-kzt9c" event={"ID":"46c568d6-e39e-4f20-89a9-376e4b667e20","Type":"ContainerStarted","Data":"7acc3d1f44071277056980debbab78e122a700cf9769e75b493010cbc421b949"} Sep 17 23:07:22.836994 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:22.836919 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2fdkb" podStartSLOduration=33.863797382 podStartE2EDuration="35.836877886s" totalImagesPullingTime="1.973080504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:22.835688113 +0000 UTC m=+70.953167600" watchObservedRunningTime="2026-09-17 23:07:22.836877886 +0000 UTC m=+70.954357344" Sep 17 23:07:22.837990 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:22.837937 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5cd8bf6458-jhsss" podStartSLOduration=33.837922543 podStartE2EDuration="33.837922543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:22.809411236 +0000 UTC m=+70.926890717" watchObservedRunningTime="2026-09-17 23:07:22.837922543 +0000 UTC m=+70.955401995" Sep 17 23:07:23.428965 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:23.428928 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:07:23.432004 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:23.431979 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:07:23.791845 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:23.791818 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:07:23.793282 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:23.793261 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5cd8bf6458-jhsss" Sep 17 23:07:24.803823 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:24.803628 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-92l5f" event={"ID":"4512e542-b8f7-4833-b2a5-9812a90f43c6","Type":"ContainerStarted","Data":"9bfc3b91cde831f76249febd5134fa611ac3923bcccc96b2b43da3f3a0e0cfac"} Sep 17 23:07:25.406136 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:25.406080 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dp5fl\" (UID: \"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:07:25.411364 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:25.411332 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5569f3-268e-4bd8-b7dc-6f5f0102f08c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dp5fl\" (UID: \"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:07:25.710324 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:25.708480 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zhqw8\"" Sep 17 23:07:25.716362 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:25.716082 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" Sep 17 23:07:32.794575 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:32.794539 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2fdkb" Sep 17 23:07:32.813872 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:32.813809 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-92l5f" podStartSLOduration=74.282482386 podStartE2EDuration="1m18.813787518s" totalImagesPullingTime="4.531305132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:24.822305848 +0000 UTC m=+72.939785301" watchObservedRunningTime="2026-09-17 23:07:32.813787518 +0000 UTC m=+80.931266964" Sep 17 23:07:33.303475 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.303396 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl"] Sep 17 23:07:33.309405 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:07:33.309347 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5569f3_268e_4bd8_b7dc_6f5f0102f08c.slice/crio-6ddaea1ec22cbb95dc9dd1a49da5075a7ae439012a4e0387ef068ae3c2a6573c WatchSource:0}: Error finding container 6ddaea1ec22cbb95dc9dd1a49da5075a7ae439012a4e0387ef068ae3c2a6573c: Status 404 returned error can't find the container with id 6ddaea1ec22cbb95dc9dd1a49da5075a7ae439012a4e0387ef068ae3c2a6573c Sep 17 23:07:33.854372 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.853796 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" event={"ID":"b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5","Type":"ContainerStarted","Data":"c3d0a09f2cc93b3482fa4f206229c122a2bb1e2fdcdd1c8be97ef54994cde205"} Sep 17 23:07:33.855749 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.855717 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" event={"ID":"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c","Type":"ContainerStarted","Data":"6ddaea1ec22cbb95dc9dd1a49da5075a7ae439012a4e0387ef068ae3c2a6573c"} Sep 17 23:07:33.859301 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.859206 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" event={"ID":"e950bcd4-51a3-45da-b5d1-f1e22867a554","Type":"ContainerStarted","Data":"56238c089b15248a123cee53721e7114b2410ed8617b50e4637e67e4bf491d7b"} Sep 17 23:07:33.859301 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.859254 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" event={"ID":"e950bcd4-51a3-45da-b5d1-f1e22867a554","Type":"ContainerStarted","Data":"c8fd394d19c8a77972374128247f81ad8ae82d3ece171e35b12cda8dcb97c9e2"} Sep 17 23:07:33.859301 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.859274 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" event={"ID":"e950bcd4-51a3-45da-b5d1-f1e22867a554","Type":"ContainerStarted","Data":"5beec7d612bcb1ae302bf2c037f40a5a849dc4aecbded5722fd6cbaacc0e18c2"} Sep 17 23:07:33.861683 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.861653 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc" event={"ID":"033e276e-f5de-4098-b9ce-73abfcee38a3","Type":"ContainerStarted","Data":"4ff59a7df98e8ed760546bf5879e479d1627ee233cbb1fa08ce2c50f7c25afdf"} Sep 17 23:07:33.863678 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.863619 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" event={"ID":"838468ce-22e4-47fd-a310-17131527ca89","Type":"ContainerStarted","Data":"0e86dbb9d059d9c25efe322640c4deb440e27515d01b4fe974b23de85fdd3831"} Sep 17 23:07:33.864138 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.864116 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:33.866129 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.866089 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" Sep 17 23:07:33.871367 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.871315 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-r2vwj" podStartSLOduration=33.746674648 podStartE2EDuration="44.871295286s" totalImagesPullingTime="11.124620638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:33.869668828 +0000 UTC m=+81.987148296" watchObservedRunningTime="2026-09-17 23:07:33.871295286 +0000 UTC m=+81.988774739" Sep 17 23:07:33.889335 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.889272 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8885b78-rc5xc" podStartSLOduration=1.756623474 podStartE2EDuration="12.889250229s" totalImagesPullingTime="11.132626755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:33.887211066 +0000 UTC m=+82.004690517" watchObservedRunningTime="2026-09-17 23:07:33.889250229 +0000 UTC m=+82.006729692" Sep 17 23:07:33.914891 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:33.914798 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59677dcbd5-9szhz" podStartSLOduration=1.790961526 podStartE2EDuration="12.914773853s" totalImagesPullingTime="11.123812327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:33.913262689 +0000 UTC m=+82.030742201" watchObservedRunningTime="2026-09-17 23:07:33.914773853 +0000 UTC m=+82.032253306" Sep 17 23:07:34.870999 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:34.870883 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" event={"ID":"8b5569f3-268e-4bd8-b7dc-6f5f0102f08c","Type":"ContainerStarted","Data":"70eaeda231157acb2a68f9e80a793b0b86061e7e31b3ef53ce416b5b484ab587"} Sep 17 23:07:34.893803 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:34.893740 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-dp5fl" podStartSLOduration=40.817795415 podStartE2EDuration="41.893719323s" totalImagesPullingTime="1.075923908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:34.8926974 +0000 UTC m=+83.010176871" watchObservedRunningTime="2026-09-17 23:07:34.893719323 +0000 UTC m=+83.011198780" Sep 17 23:07:34.895172 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:34.894957 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cb9cb6f5-h7svf" podStartSLOduration=2.7765320300000003 podStartE2EDuration="13.894939414s" totalImagesPullingTime="11.118407384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:33.949305929 +0000 UTC m=+82.066785382" watchObservedRunningTime="2026-09-17 23:07:34.894939414 +0000 UTC m=+83.012418866" Sep 17 23:07:39.305603 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:39.305553 2559 patch_prober.go:28] interesting pod/image-registry-7f64f764df-gtws9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 17 23:07:39.306189 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:39.305628 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" podUID="ce53708a-abfa-4b74-b949-2456ce5cd297" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 23:07:40.768052 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:40.768023 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:07:42.914011 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:42.913958 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-kzt9c" event={"ID":"46c568d6-e39e-4f20-89a9-376e4b667e20","Type":"ContainerStarted","Data":"80da483bc4e892ef318e9e1eb1ec9205fc2407c2d3bc4d0ded84adcab2b86b40"} Sep 17 23:07:42.914486 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:42.914257 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-kzt9c" Sep 17 23:07:42.931148 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:42.931116 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-kzt9c" Sep 17 23:07:42.938810 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:42.938753 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-kzt9c" podStartSLOduration=1.490707777 podStartE2EDuration="21.938731821s" totalImagesPullingTime="20.448024044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:42.937057884 +0000 UTC m=+91.054537360" watchObservedRunningTime="2026-09-17 23:07:42.938731821 +0000 UTC m=+91.056211273" Sep 17 23:07:44.087607 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.087568 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-njz52"] Sep 17 23:07:44.113226 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.113194 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.115725 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.115692 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 23:07:44.118123 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.118095 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 23:07:44.118263 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.118151 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 23:07:44.118263 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.118171 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5h2qn\"" Sep 17 23:07:44.118667 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.118647 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 23:07:44.180888 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.180847 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-accelerators-collector-config\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.181083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.180915 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-wtmp\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.181083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.180998 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-tls\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.181083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.181049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-textfile\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.181083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.181081 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6351ced2-25b1-44c2-9ceb-17294d942b5c-sys\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.181285 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.181109 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.181285 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.181135 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6blhl\" (UniqueName: \"kubernetes.io/projected/6351ced2-25b1-44c2-9ceb-17294d942b5c-kube-api-access-6blhl\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.181285 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.181180 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6351ced2-25b1-44c2-9ceb-17294d942b5c-metrics-client-ca\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.181285 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.181220 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6351ced2-25b1-44c2-9ceb-17294d942b5c-root\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.283752 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.283708 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6351ced2-25b1-44c2-9ceb-17294d942b5c-metrics-client-ca\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.283958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.283778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6351ced2-25b1-44c2-9ceb-17294d942b5c-root\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.283958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.283826 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-accelerators-collector-config\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.283958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.283864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-wtmp\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.283958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.283925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-tls\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.284190 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.283965 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-textfile\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.284190 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.283994 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6351ced2-25b1-44c2-9ceb-17294d942b5c-sys\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.284190 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.284034 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.284190 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.284062 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6blhl\" (UniqueName: \"kubernetes.io/projected/6351ced2-25b1-44c2-9ceb-17294d942b5c-kube-api-access-6blhl\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.285116 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.285088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6351ced2-25b1-44c2-9ceb-17294d942b5c-metrics-client-ca\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.285267 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.285176 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6351ced2-25b1-44c2-9ceb-17294d942b5c-root\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.285683 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.285661 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-accelerators-collector-config\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.285834 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.285816 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-wtmp\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.286148 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.286128 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6351ced2-25b1-44c2-9ceb-17294d942b5c-sys\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.287625 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.286657 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-textfile\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.290402 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.290378 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-tls\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.290593 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.290518 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6351ced2-25b1-44c2-9ceb-17294d942b5c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.294029 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.294004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6blhl\" (UniqueName: \"kubernetes.io/projected/6351ced2-25b1-44c2-9ceb-17294d942b5c-kube-api-access-6blhl\") pod \"node-exporter-njz52\" (UID: \"6351ced2-25b1-44c2-9ceb-17294d942b5c\") " pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.425755 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.425673 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-njz52" Sep 17 23:07:44.803323 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.803290 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f64f764df-gtws9"] Sep 17 23:07:44.927105 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:44.927063 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njz52" event={"ID":"6351ced2-25b1-44c2-9ceb-17294d942b5c","Type":"ContainerStarted","Data":"0696749d0e6eb7035578d413870eedb4e853e46dcdda8ee8bc1f9efce392183c"} Sep 17 23:07:46.940058 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:46.940013 2559 generic.go:356] "Generic (PLEG): container finished" podID="6351ced2-25b1-44c2-9ceb-17294d942b5c" containerID="06cbfc05eb9a04e049f887dbf3dbea0b9458093af12557709c40c55f78916879" exitCode=0 Sep 17 23:07:46.940543 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:46.940065 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njz52" event={"ID":"6351ced2-25b1-44c2-9ceb-17294d942b5c","Type":"ContainerDied","Data":"06cbfc05eb9a04e049f887dbf3dbea0b9458093af12557709c40c55f78916879"} Sep 17 23:07:47.948235 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:47.948191 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njz52" event={"ID":"6351ced2-25b1-44c2-9ceb-17294d942b5c","Type":"ContainerStarted","Data":"a0763ef18882319b80b45729d6e488665c91c573a686929a86e332a375804bbe"} Sep 17 23:07:47.948668 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:47.948249 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njz52" event={"ID":"6351ced2-25b1-44c2-9ceb-17294d942b5c","Type":"ContainerStarted","Data":"3ddd54f2866947be22733469cc913d22583514fe3830d62f2446ccdc53e1ec78"} Sep 17 23:07:47.967823 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:47.967767 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-njz52" podStartSLOduration=2.480494934 podStartE2EDuration="3.967748486s" totalImagesPullingTime="1.390968676s" totalInitContainerRuntime="96.284876ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:47.967325796 +0000 UTC m=+96.084805266" watchObservedRunningTime="2026-09-17 23:07:47.967748486 +0000 UTC m=+96.085227937" Sep 17 23:07:50.765939 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:07:50.765824 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sq8mp" Sep 17 23:08:09.034556 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:09.034522 2559 generic.go:356] "Generic (PLEG): container finished" podID="38d9d121-59fe-483b-a6c4-f29510ab8230" containerID="e4ad10a0506bc34f35966a8ab17301555710342fa413d71800525276065d24b6" exitCode=0 Sep 17 23:08:09.035048 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:09.034604 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" event={"ID":"38d9d121-59fe-483b-a6c4-f29510ab8230","Type":"ContainerDied","Data":"e4ad10a0506bc34f35966a8ab17301555710342fa413d71800525276065d24b6"} Sep 17 23:08:09.035112 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:09.035059 2559 scope.go:117] "RemoveContainer" containerID="e4ad10a0506bc34f35966a8ab17301555710342fa413d71800525276065d24b6" Sep 17 23:08:09.036162 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:09.036141 2559 generic.go:356] "Generic (PLEG): container finished" podID="a9479c5a-db24-4df7-9402-b6dffa701fa4" containerID="20ab9191f293494f126fa22f76c679bebc68ffa34d174d04463e98b82f3bd8a3" exitCode=0 Sep 17 23:08:09.036247 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:09.036169 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" event={"ID":"a9479c5a-db24-4df7-9402-b6dffa701fa4","Type":"ContainerDied","Data":"20ab9191f293494f126fa22f76c679bebc68ffa34d174d04463e98b82f3bd8a3"} Sep 17 23:08:09.036461 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:09.036449 2559 scope.go:117] "RemoveContainer" containerID="20ab9191f293494f126fa22f76c679bebc68ffa34d174d04463e98b82f3bd8a3" Sep 17 23:08:09.827079 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:09.827037 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" podUID="ce53708a-abfa-4b74-b949-2456ce5cd297" containerName="registry" containerID="cri-o://bba548b94e8251e0de609cd20b8479f79c631c301e3efa879ce19cb6a4c9e7fa" gracePeriod=30 Sep 17 23:08:10.042984 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.042953 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce53708a-abfa-4b74-b949-2456ce5cd297" containerID="bba548b94e8251e0de609cd20b8479f79c631c301e3efa879ce19cb6a4c9e7fa" exitCode=0 Sep 17 23:08:10.043441 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.043022 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" event={"ID":"ce53708a-abfa-4b74-b949-2456ce5cd297","Type":"ContainerDied","Data":"bba548b94e8251e0de609cd20b8479f79c631c301e3efa879ce19cb6a4c9e7fa"} Sep 17 23:08:10.045178 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.045146 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-g6ngg" event={"ID":"38d9d121-59fe-483b-a6c4-f29510ab8230","Type":"ContainerStarted","Data":"b87b2b36e93fd7c13000776f5884ac8aba0b9a08548db76dc9be60e212a6c637"} Sep 17 23:08:10.047415 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.047390 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4gmns" event={"ID":"a9479c5a-db24-4df7-9402-b6dffa701fa4","Type":"ContainerStarted","Data":"4c90fb25e7c8e29f0d7bcea070a3a4ddf784602cc623edab351f0f17afdb100c"} Sep 17 23:08:10.137310 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.137275 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:08:10.313821 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.313779 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls\") pod \"ce53708a-abfa-4b74-b949-2456ce5cd297\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " Sep 17 23:08:10.313821 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.313817 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce53708a-abfa-4b74-b949-2456ce5cd297-installation-pull-secrets\") pod \"ce53708a-abfa-4b74-b949-2456ce5cd297\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " Sep 17 23:08:10.314083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.313851 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce53708a-abfa-4b74-b949-2456ce5cd297-image-registry-private-configuration\") pod \"ce53708a-abfa-4b74-b949-2456ce5cd297\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " Sep 17 23:08:10.314083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.313935 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-certificates\") pod \"ce53708a-abfa-4b74-b949-2456ce5cd297\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " Sep 17 23:08:10.314083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.313990 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjlqp\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-kube-api-access-sjlqp\") pod \"ce53708a-abfa-4b74-b949-2456ce5cd297\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " Sep 17 23:08:10.314083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.314020 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce53708a-abfa-4b74-b949-2456ce5cd297-trusted-ca\") pod \"ce53708a-abfa-4b74-b949-2456ce5cd297\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " Sep 17 23:08:10.314083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.314053 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-bound-sa-token\") pod \"ce53708a-abfa-4b74-b949-2456ce5cd297\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " Sep 17 23:08:10.314336 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.314089 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce53708a-abfa-4b74-b949-2456ce5cd297-ca-trust-extracted\") pod \"ce53708a-abfa-4b74-b949-2456ce5cd297\" (UID: \"ce53708a-abfa-4b74-b949-2456ce5cd297\") " Sep 17 23:08:10.314648 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.314616 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ce53708a-abfa-4b74-b949-2456ce5cd297" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:08:10.314719 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.314691 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce53708a-abfa-4b74-b949-2456ce5cd297-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ce53708a-abfa-4b74-b949-2456ce5cd297" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:08:10.316864 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.316738 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce53708a-abfa-4b74-b949-2456ce5cd297-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ce53708a-abfa-4b74-b949-2456ce5cd297" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:08:10.316864 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.316812 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ce53708a-abfa-4b74-b949-2456ce5cd297" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:08:10.316864 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.316853 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce53708a-abfa-4b74-b949-2456ce5cd297-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ce53708a-abfa-4b74-b949-2456ce5cd297" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:08:10.317137 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.317046 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ce53708a-abfa-4b74-b949-2456ce5cd297" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:08:10.317195 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.317157 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-kube-api-access-sjlqp" (OuterVolumeSpecName: "kube-api-access-sjlqp") pod "ce53708a-abfa-4b74-b949-2456ce5cd297" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297"). InnerVolumeSpecName "kube-api-access-sjlqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:08:10.324089 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.324061 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce53708a-abfa-4b74-b949-2456ce5cd297-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ce53708a-abfa-4b74-b949-2456ce5cd297" (UID: "ce53708a-abfa-4b74-b949-2456ce5cd297"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:08:10.415467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.415367 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-certificates\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:08:10.415467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.415404 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjlqp\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-kube-api-access-sjlqp\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:08:10.415467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.415417 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce53708a-abfa-4b74-b949-2456ce5cd297-trusted-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:08:10.415467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.415429 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-bound-sa-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:08:10.415467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.415443 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce53708a-abfa-4b74-b949-2456ce5cd297-ca-trust-extracted\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:08:10.415467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.415456 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce53708a-abfa-4b74-b949-2456ce5cd297-registry-tls\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:08:10.415467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.415469 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce53708a-abfa-4b74-b949-2456ce5cd297-installation-pull-secrets\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:08:10.415790 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:10.415482 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce53708a-abfa-4b74-b949-2456ce5cd297-image-registry-private-configuration\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:08:11.053239 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:11.053203 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" event={"ID":"ce53708a-abfa-4b74-b949-2456ce5cd297","Type":"ContainerDied","Data":"fc73ef36871523f7b1e7a142a16335787e68022c0bf1a07ed38f5be6fa58d518"} Sep 17 23:08:11.053656 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:11.053262 2559 scope.go:117] "RemoveContainer" containerID="bba548b94e8251e0de609cd20b8479f79c631c301e3efa879ce19cb6a4c9e7fa" Sep 17 23:08:11.053656 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:11.053219 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f64f764df-gtws9" Sep 17 23:08:11.076300 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:11.076266 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f64f764df-gtws9"] Sep 17 23:08:11.080977 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:11.080948 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f64f764df-gtws9"] Sep 17 23:08:12.290304 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:12.288132 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce53708a-abfa-4b74-b949-2456ce5cd297" path="/var/lib/kubelet/pods/ce53708a-abfa-4b74-b949-2456ce5cd297/volumes" Sep 17 23:08:24.106397 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:24.106361 2559 generic.go:356] "Generic (PLEG): container finished" podID="555eaa00-e5cb-49e7-adbb-18ccfa293242" containerID="44902ceb92f25df18fedc8a3cb773d5a4c5b84d4175915d16b13c99be58e5560" exitCode=0 Sep 17 23:08:24.106920 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:24.106415 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-gw259" event={"ID":"555eaa00-e5cb-49e7-adbb-18ccfa293242","Type":"ContainerDied","Data":"44902ceb92f25df18fedc8a3cb773d5a4c5b84d4175915d16b13c99be58e5560"} Sep 17 23:08:24.106920 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:24.106863 2559 scope.go:117] "RemoveContainer" containerID="44902ceb92f25df18fedc8a3cb773d5a4c5b84d4175915d16b13c99be58e5560" Sep 17 23:08:25.112697 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:08:25.112663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-gw259" event={"ID":"555eaa00-e5cb-49e7-adbb-18ccfa293242","Type":"ContainerStarted","Data":"96321a8baf77e17278ff19fe74de19a8f81c761328790fbb1840388e733299fc"} Sep 17 23:11:09.273480 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.273448 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p"] Sep 17 23:11:09.273920 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.273787 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce53708a-abfa-4b74-b949-2456ce5cd297" containerName="registry" Sep 17 23:11:09.273920 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.273799 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce53708a-abfa-4b74-b949-2456ce5cd297" containerName="registry" Sep 17 23:11:09.273920 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.273851 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce53708a-abfa-4b74-b949-2456ce5cd297" containerName="registry" Sep 17 23:11:09.275689 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.275674 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.278330 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.278305 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 17 23:11:09.278462 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.278340 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 17 23:11:09.278462 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.278389 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-czjhc\"" Sep 17 23:11:09.278717 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.278700 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 17 23:11:09.278775 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.278759 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 17 23:11:09.295982 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.295956 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p"] Sep 17 23:11:09.334119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.334071 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b629004d-5aee-4574-89b8-6e63b6e968c8-manifests\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.334119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.334123 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b629004d-5aee-4574-89b8-6e63b6e968c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.334329 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.334147 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b629004d-5aee-4574-89b8-6e63b6e968c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.334329 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.334170 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b629004d-5aee-4574-89b8-6e63b6e968c8-tmp\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.334329 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.334262 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2lt\" (UniqueName: \"kubernetes.io/projected/b629004d-5aee-4574-89b8-6e63b6e968c8-kube-api-access-bf2lt\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.434783 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.434748 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b629004d-5aee-4574-89b8-6e63b6e968c8-manifests\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.434783 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.434784 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b629004d-5aee-4574-89b8-6e63b6e968c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.435074 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.434808 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b629004d-5aee-4574-89b8-6e63b6e968c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.435074 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.434829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b629004d-5aee-4574-89b8-6e63b6e968c8-tmp\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.435074 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.434881 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2lt\" (UniqueName: \"kubernetes.io/projected/b629004d-5aee-4574-89b8-6e63b6e968c8-kube-api-access-bf2lt\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.435227 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.435158 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b629004d-5aee-4574-89b8-6e63b6e968c8-manifests\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.437155 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.437130 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b629004d-5aee-4574-89b8-6e63b6e968c8-tmp\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.437512 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.437490 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b629004d-5aee-4574-89b8-6e63b6e968c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.437553 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.437490 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b629004d-5aee-4574-89b8-6e63b6e968c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.443620 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.443594 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2lt\" (UniqueName: \"kubernetes.io/projected/b629004d-5aee-4574-89b8-6e63b6e968c8-kube-api-access-bf2lt\") pod \"opendatahub-operator-controller-manager-697bb5576d-qqc9p\" (UID: \"b629004d-5aee-4574-89b8-6e63b6e968c8\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.585766 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.585686 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:09.860612 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:09.860580 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p"] Sep 17 23:11:09.862672 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:11:09.862636 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb629004d_5aee_4574_89b8_6e63b6e968c8.slice/crio-e92a90dbfad6510e48a1220f4da07541368dbb01ada469fc6c64779cc5b5799d WatchSource:0}: Error finding container e92a90dbfad6510e48a1220f4da07541368dbb01ada469fc6c64779cc5b5799d: Status 404 returned error can't find the container with id e92a90dbfad6510e48a1220f4da07541368dbb01ada469fc6c64779cc5b5799d Sep 17 23:11:10.775190 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:10.775148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" event={"ID":"b629004d-5aee-4574-89b8-6e63b6e968c8","Type":"ContainerStarted","Data":"e92a90dbfad6510e48a1220f4da07541368dbb01ada469fc6c64779cc5b5799d"} Sep 17 23:11:12.786993 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:12.786955 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" event={"ID":"b629004d-5aee-4574-89b8-6e63b6e968c8","Type":"ContainerStarted","Data":"9d95baf975d00fb0494bb55c4fbdb52124e77dfa19dbb2e3f13f67f418c3b4fd"} Sep 17 23:11:13.792437 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:13.792402 2559 generic.go:356] "Generic (PLEG): container finished" podID="b629004d-5aee-4574-89b8-6e63b6e968c8" containerID="9d95baf975d00fb0494bb55c4fbdb52124e77dfa19dbb2e3f13f67f418c3b4fd" exitCode=0 Sep 17 23:11:13.792896 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:13.792434 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" event={"ID":"b629004d-5aee-4574-89b8-6e63b6e968c8","Type":"ContainerDied","Data":"9d95baf975d00fb0494bb55c4fbdb52124e77dfa19dbb2e3f13f67f418c3b4fd"} Sep 17 23:11:14.154518 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:14.154501 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:11:14.157598 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:14.157574 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:11:14.168251 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:14.168233 2559 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 23:11:14.798560 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:14.798524 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" event={"ID":"b629004d-5aee-4574-89b8-6e63b6e968c8","Type":"ContainerStarted","Data":"f55ed42a61cc981993981e996d2dc7053dad8ba3187f974b27040dc353c0095e"} Sep 17 23:11:14.798984 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:14.798663 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:14.840527 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:14.840441 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" podStartSLOduration=1.776022748 podStartE2EDuration="5.840424338s" totalImagesPullingTime="3.03783987s" totalInitContainerRuntime="1.02656172s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:14.838373434 +0000 UTC m=+302.955852912" watchObservedRunningTime="2026-09-17 23:11:14.840424338 +0000 UTC m=+302.957903789" Sep 17 23:11:25.805631 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:25.805593 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-qqc9p" Sep 17 23:11:28.552331 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.552290 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q"] Sep 17 23:11:28.557444 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.557413 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" Sep 17 23:11:28.559868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.559843 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 17 23:11:28.560018 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.559873 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 17 23:11:28.573542 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.573516 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q"] Sep 17 23:11:28.697261 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.697223 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94540370-de05-4b11-8010-5becb5180325-tmp\") pod \"kube-auth-proxy-65f4688f79-d4v4q\" (UID: \"94540370-de05-4b11-8010-5becb5180325\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" Sep 17 23:11:28.697261 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.697263 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2jnv\" (UniqueName: \"kubernetes.io/projected/94540370-de05-4b11-8010-5becb5180325-kube-api-access-h2jnv\") pod \"kube-auth-proxy-65f4688f79-d4v4q\" (UID: \"94540370-de05-4b11-8010-5becb5180325\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" Sep 17 23:11:28.697464 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.697289 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94540370-de05-4b11-8010-5becb5180325-tls-certs\") pod \"kube-auth-proxy-65f4688f79-d4v4q\" (UID: \"94540370-de05-4b11-8010-5becb5180325\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" Sep 17 23:11:28.798131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.798091 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94540370-de05-4b11-8010-5becb5180325-tmp\") pod \"kube-auth-proxy-65f4688f79-d4v4q\" (UID: \"94540370-de05-4b11-8010-5becb5180325\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" Sep 17 23:11:28.798131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.798132 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2jnv\" (UniqueName: \"kubernetes.io/projected/94540370-de05-4b11-8010-5becb5180325-kube-api-access-h2jnv\") pod \"kube-auth-proxy-65f4688f79-d4v4q\" (UID: \"94540370-de05-4b11-8010-5becb5180325\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" Sep 17 23:11:28.798328 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.798151 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94540370-de05-4b11-8010-5becb5180325-tls-certs\") pod \"kube-auth-proxy-65f4688f79-d4v4q\" (UID: \"94540370-de05-4b11-8010-5becb5180325\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" Sep 17 23:11:28.800434 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.800409 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94540370-de05-4b11-8010-5becb5180325-tmp\") pod \"kube-auth-proxy-65f4688f79-d4v4q\" (UID: \"94540370-de05-4b11-8010-5becb5180325\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" Sep 17 23:11:28.800626 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.800595 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94540370-de05-4b11-8010-5becb5180325-tls-certs\") pod \"kube-auth-proxy-65f4688f79-d4v4q\" (UID: \"94540370-de05-4b11-8010-5becb5180325\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" Sep 17 23:11:28.812694 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.812617 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2jnv\" (UniqueName: \"kubernetes.io/projected/94540370-de05-4b11-8010-5becb5180325-kube-api-access-h2jnv\") pod \"kube-auth-proxy-65f4688f79-d4v4q\" (UID: \"94540370-de05-4b11-8010-5becb5180325\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" Sep 17 23:11:28.867619 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:28.867587 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" Sep 17 23:11:29.145182 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:29.145100 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q"] Sep 17 23:11:29.149252 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:11:29.149218 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94540370_de05_4b11_8010_5becb5180325.slice/crio-0809ff85fdaa0150e16f1942e9da904964cb5c7712bf3fbd8c62b732571e80de WatchSource:0}: Error finding container 0809ff85fdaa0150e16f1942e9da904964cb5c7712bf3fbd8c62b732571e80de: Status 404 returned error can't find the container with id 0809ff85fdaa0150e16f1942e9da904964cb5c7712bf3fbd8c62b732571e80de Sep 17 23:11:29.151114 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:29.151090 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 23:11:29.870617 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:29.870564 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" event={"ID":"94540370-de05-4b11-8010-5becb5180325","Type":"ContainerStarted","Data":"0809ff85fdaa0150e16f1942e9da904964cb5c7712bf3fbd8c62b732571e80de"} Sep 17 23:11:30.015517 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.015484 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc"] Sep 17 23:11:30.018307 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.018267 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.020934 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.020889 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 17 23:11:30.021184 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.021163 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 17 23:11:30.021327 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.021251 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 23:11:30.021405 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.021053 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xfhng\"" Sep 17 23:11:30.021405 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.021033 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 17 23:11:30.031668 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.031590 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc"] Sep 17 23:11:30.109063 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.109023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfce8216-c86e-40e5-93de-98bf1a83ea4a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.109226 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.109092 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.109226 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.109134 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdn9\" (UniqueName: \"kubernetes.io/projected/bfce8216-c86e-40e5-93de-98bf1a83ea4a-kube-api-access-xrdn9\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.109226 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.109209 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.109465 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.109286 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.109465 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.109334 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.109465 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.109375 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.210213 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.210181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.210385 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.210223 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.210385 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.210258 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfce8216-c86e-40e5-93de-98bf1a83ea4a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.210385 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.210300 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.210385 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.210336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdn9\" (UniqueName: \"kubernetes.io/projected/bfce8216-c86e-40e5-93de-98bf1a83ea4a-kube-api-access-xrdn9\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.210385 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.210377 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.210648 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.210413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.211354 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.211324 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.213190 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.213158 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.213524 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.213499 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.213722 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.213703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfce8216-c86e-40e5-93de-98bf1a83ea4a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.213952 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.213932 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.224283 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.224262 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.224840 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.224813 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdn9\" (UniqueName: \"kubernetes.io/projected/bfce8216-c86e-40e5-93de-98bf1a83ea4a-kube-api-access-xrdn9\") pod \"istiod-openshift-gateway-7b7995d5cd-vtckc\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:30.332352 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:30.332322 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:31.842331 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:31.842253 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc"] Sep 17 23:11:31.845475 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:11:31.845446 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfce8216_c86e_40e5_93de_98bf1a83ea4a.slice/crio-b4c92e561abb20ed8dfd7a9f1e29f2aff1209ca50e6552f4cd595e993eaf7caa WatchSource:0}: Error finding container b4c92e561abb20ed8dfd7a9f1e29f2aff1209ca50e6552f4cd595e993eaf7caa: Status 404 returned error can't find the container with id b4c92e561abb20ed8dfd7a9f1e29f2aff1209ca50e6552f4cd595e993eaf7caa Sep 17 23:11:31.881607 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:31.881571 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" event={"ID":"bfce8216-c86e-40e5-93de-98bf1a83ea4a","Type":"ContainerStarted","Data":"b4c92e561abb20ed8dfd7a9f1e29f2aff1209ca50e6552f4cd595e993eaf7caa"} Sep 17 23:11:31.883084 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:31.883051 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" event={"ID":"94540370-de05-4b11-8010-5becb5180325","Type":"ContainerStarted","Data":"bd4ff24bc74cb9ca250888d504260f0ac652e288864498f907684c65d0d19247"} Sep 17 23:11:31.902537 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:31.902481 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65f4688f79-d4v4q" podStartSLOduration=1.453533083 podStartE2EDuration="3.902462728s" totalImagesPullingTime="2.448929645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:31.902109323 +0000 UTC m=+320.019588791" watchObservedRunningTime="2026-09-17 23:11:31.902462728 +0000 UTC m=+320.019942177" Sep 17 23:11:33.945646 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:33.945608 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:11:33.945887 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:33.945677 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:11:34.901887 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:34.901835 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" event={"ID":"bfce8216-c86e-40e5-93de-98bf1a83ea4a","Type":"ContainerStarted","Data":"19508f95523088e36a76440e545715e01f4ed404bedd591e7d1bc2827e649a35"} Sep 17 23:11:34.902080 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:34.902027 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:34.903640 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:34.903614 2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vtckc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 23:11:34.903761 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:34.903667 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" podUID="bfce8216-c86e-40e5-93de-98bf1a83ea4a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 23:11:34.944522 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:34.944467 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" podStartSLOduration=3.846497526 podStartE2EDuration="5.94445131s" totalImagesPullingTime="2.097953784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:34.942784254 +0000 UTC m=+323.060263730" watchObservedRunningTime="2026-09-17 23:11:34.94445131 +0000 UTC m=+323.061930759" Sep 17 23:11:35.907312 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:35.907282 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:48.826216 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.826182 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw"] Sep 17 23:11:48.829471 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.829449 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" Sep 17 23:11:48.831693 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.831673 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 17 23:11:48.831812 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.831724 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-897tm\"" Sep 17 23:11:48.838707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.838684 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm"] Sep 17 23:11:48.841444 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.841421 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" Sep 17 23:11:48.841939 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.841916 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw"] Sep 17 23:11:48.844100 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.844080 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 17 23:11:48.844197 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.844080 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-sdgq7\"" Sep 17 23:11:48.853066 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.852804 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm"] Sep 17 23:11:48.874187 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.874155 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c9c9334-c345-4bfd-9977-814550d49937-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t7ncm\" (UID: \"8c9c9334-c345-4bfd-9977-814550d49937\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" Sep 17 23:11:48.874310 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.874220 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a39c0513-f908-4653-8ae0-2cb0aad85b34-cert\") pod \"kserve-controller-manager-d6f4b65b8-z7jjw\" (UID: \"a39c0513-f908-4653-8ae0-2cb0aad85b34\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" Sep 17 23:11:48.874354 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.874330 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blwpk\" (UniqueName: \"kubernetes.io/projected/a39c0513-f908-4653-8ae0-2cb0aad85b34-kube-api-access-blwpk\") pod \"kserve-controller-manager-d6f4b65b8-z7jjw\" (UID: \"a39c0513-f908-4653-8ae0-2cb0aad85b34\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" Sep 17 23:11:48.874450 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.874432 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2h97\" (UniqueName: \"kubernetes.io/projected/8c9c9334-c345-4bfd-9977-814550d49937-kube-api-access-k2h97\") pod \"llmisvc-controller-manager-777c6bd7d5-t7ncm\" (UID: \"8c9c9334-c345-4bfd-9977-814550d49937\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" Sep 17 23:11:48.975574 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.975534 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2h97\" (UniqueName: \"kubernetes.io/projected/8c9c9334-c345-4bfd-9977-814550d49937-kube-api-access-k2h97\") pod \"llmisvc-controller-manager-777c6bd7d5-t7ncm\" (UID: \"8c9c9334-c345-4bfd-9977-814550d49937\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" Sep 17 23:11:48.975574 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.975578 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c9c9334-c345-4bfd-9977-814550d49937-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t7ncm\" (UID: \"8c9c9334-c345-4bfd-9977-814550d49937\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" Sep 17 23:11:48.975826 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.975633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a39c0513-f908-4653-8ae0-2cb0aad85b34-cert\") pod \"kserve-controller-manager-d6f4b65b8-z7jjw\" (UID: \"a39c0513-f908-4653-8ae0-2cb0aad85b34\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" Sep 17 23:11:48.975826 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.975670 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blwpk\" (UniqueName: \"kubernetes.io/projected/a39c0513-f908-4653-8ae0-2cb0aad85b34-kube-api-access-blwpk\") pod \"kserve-controller-manager-d6f4b65b8-z7jjw\" (UID: \"a39c0513-f908-4653-8ae0-2cb0aad85b34\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" Sep 17 23:11:48.978287 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.978255 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a39c0513-f908-4653-8ae0-2cb0aad85b34-cert\") pod \"kserve-controller-manager-d6f4b65b8-z7jjw\" (UID: \"a39c0513-f908-4653-8ae0-2cb0aad85b34\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" Sep 17 23:11:48.978419 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.978358 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c9c9334-c345-4bfd-9977-814550d49937-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t7ncm\" (UID: \"8c9c9334-c345-4bfd-9977-814550d49937\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" Sep 17 23:11:48.997349 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.997315 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2h97\" (UniqueName: \"kubernetes.io/projected/8c9c9334-c345-4bfd-9977-814550d49937-kube-api-access-k2h97\") pod \"llmisvc-controller-manager-777c6bd7d5-t7ncm\" (UID: \"8c9c9334-c345-4bfd-9977-814550d49937\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" Sep 17 23:11:48.999649 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:48.999627 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blwpk\" (UniqueName: \"kubernetes.io/projected/a39c0513-f908-4653-8ae0-2cb0aad85b34-kube-api-access-blwpk\") pod \"kserve-controller-manager-d6f4b65b8-z7jjw\" (UID: \"a39c0513-f908-4653-8ae0-2cb0aad85b34\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" Sep 17 23:11:49.142834 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:49.142752 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" Sep 17 23:11:49.154016 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:49.153985 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" Sep 17 23:11:49.415520 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:49.415445 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw"] Sep 17 23:11:49.418820 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:11:49.418793 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39c0513_f908_4653_8ae0_2cb0aad85b34.slice/crio-78c2cd6b0a58e857239f15b445bce141fa56b8d7b22554fb00e28bd51928a28c WatchSource:0}: Error finding container 78c2cd6b0a58e857239f15b445bce141fa56b8d7b22554fb00e28bd51928a28c: Status 404 returned error can't find the container with id 78c2cd6b0a58e857239f15b445bce141fa56b8d7b22554fb00e28bd51928a28c Sep 17 23:11:49.433199 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:49.433137 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm"] Sep 17 23:11:49.435429 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:11:49.435404 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8c9c9334_c345_4bfd_9977_814550d49937.slice/crio-30f408df72740efaf1ce24e0ab8a23ce7ea1e6e7bac7d3c5ec9be94c11708964 WatchSource:0}: Error finding container 30f408df72740efaf1ce24e0ab8a23ce7ea1e6e7bac7d3c5ec9be94c11708964: Status 404 returned error can't find the container with id 30f408df72740efaf1ce24e0ab8a23ce7ea1e6e7bac7d3c5ec9be94c11708964 Sep 17 23:11:49.973620 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:49.973573 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" event={"ID":"a39c0513-f908-4653-8ae0-2cb0aad85b34","Type":"ContainerStarted","Data":"78c2cd6b0a58e857239f15b445bce141fa56b8d7b22554fb00e28bd51928a28c"} Sep 17 23:11:49.976396 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:49.976321 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" event={"ID":"8c9c9334-c345-4bfd-9977-814550d49937","Type":"ContainerStarted","Data":"30f408df72740efaf1ce24e0ab8a23ce7ea1e6e7bac7d3c5ec9be94c11708964"} Sep 17 23:11:50.985054 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:50.985024 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc"] Sep 17 23:11:50.985471 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:50.985244 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" podUID="bfce8216-c86e-40e5-93de-98bf1a83ea4a" containerName="discovery" containerID="cri-o://19508f95523088e36a76440e545715e01f4ed404bedd591e7d1bc2827e649a35" gracePeriod=30 Sep 17 23:11:51.989565 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:51.989521 2559 generic.go:356] "Generic (PLEG): container finished" podID="bfce8216-c86e-40e5-93de-98bf1a83ea4a" containerID="19508f95523088e36a76440e545715e01f4ed404bedd591e7d1bc2827e649a35" exitCode=0 Sep 17 23:11:51.990050 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:51.989575 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" event={"ID":"bfce8216-c86e-40e5-93de-98bf1a83ea4a","Type":"ContainerDied","Data":"19508f95523088e36a76440e545715e01f4ed404bedd591e7d1bc2827e649a35"} Sep 17 23:11:52.106351 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.106321 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:52.206859 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.206826 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-kubeconfig\") pod \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " Sep 17 23:11:52.207081 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.206972 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-csr-ca-configmap\") pod \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " Sep 17 23:11:52.207081 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.207003 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-token\") pod \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " Sep 17 23:11:52.207081 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.207036 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-csr-dns-cert\") pod \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " Sep 17 23:11:52.207081 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.207062 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-cacerts\") pod \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " Sep 17 23:11:52.207291 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.207102 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfce8216-c86e-40e5-93de-98bf1a83ea4a-local-certs\") pod \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " Sep 17 23:11:52.207291 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.207145 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrdn9\" (UniqueName: \"kubernetes.io/projected/bfce8216-c86e-40e5-93de-98bf1a83ea4a-kube-api-access-xrdn9\") pod \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\" (UID: \"bfce8216-c86e-40e5-93de-98bf1a83ea4a\") " Sep 17 23:11:52.207396 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.207362 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "bfce8216-c86e-40e5-93de-98bf1a83ea4a" (UID: "bfce8216-c86e-40e5-93de-98bf1a83ea4a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:11:52.209380 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.209350 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-cacerts" (OuterVolumeSpecName: "cacerts") pod "bfce8216-c86e-40e5-93de-98bf1a83ea4a" (UID: "bfce8216-c86e-40e5-93de-98bf1a83ea4a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:11:52.209668 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.209647 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "bfce8216-c86e-40e5-93de-98bf1a83ea4a" (UID: "bfce8216-c86e-40e5-93de-98bf1a83ea4a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:11:52.209758 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.209669 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-token" (OuterVolumeSpecName: "istio-token") pod "bfce8216-c86e-40e5-93de-98bf1a83ea4a" (UID: "bfce8216-c86e-40e5-93de-98bf1a83ea4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:11:52.209840 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.209818 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfce8216-c86e-40e5-93de-98bf1a83ea4a-kube-api-access-xrdn9" (OuterVolumeSpecName: "kube-api-access-xrdn9") pod "bfce8216-c86e-40e5-93de-98bf1a83ea4a" (UID: "bfce8216-c86e-40e5-93de-98bf1a83ea4a"). InnerVolumeSpecName "kube-api-access-xrdn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:11:52.240595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.240519 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "bfce8216-c86e-40e5-93de-98bf1a83ea4a" (UID: "bfce8216-c86e-40e5-93de-98bf1a83ea4a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:11:52.240595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.240540 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfce8216-c86e-40e5-93de-98bf1a83ea4a-local-certs" (OuterVolumeSpecName: "local-certs") pod "bfce8216-c86e-40e5-93de-98bf1a83ea4a" (UID: "bfce8216-c86e-40e5-93de-98bf1a83ea4a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:11:52.308682 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.308658 2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-csr-ca-configmap\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:11:52.308682 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.308680 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:11:52.308838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.308691 2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-csr-dns-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:11:52.308838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.308700 2559 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-cacerts\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:11:52.308838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.308708 2559 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfce8216-c86e-40e5-93de-98bf1a83ea4a-local-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:11:52.308838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.308717 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrdn9\" (UniqueName: \"kubernetes.io/projected/bfce8216-c86e-40e5-93de-98bf1a83ea4a-kube-api-access-xrdn9\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:11:52.308838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.308725 2559 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfce8216-c86e-40e5-93de-98bf1a83ea4a-istio-kubeconfig\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:11:52.996013 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.995975 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" event={"ID":"a39c0513-f908-4653-8ae0-2cb0aad85b34","Type":"ContainerStarted","Data":"c0598362629cf3505512fe2619a486939958a4e710b91eccdbbd55cdac717783"} Sep 17 23:11:52.996430 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.996100 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" Sep 17 23:11:52.997484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.997468 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" Sep 17 23:11:52.997484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.997472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc" event={"ID":"bfce8216-c86e-40e5-93de-98bf1a83ea4a","Type":"ContainerDied","Data":"b4c92e561abb20ed8dfd7a9f1e29f2aff1209ca50e6552f4cd595e993eaf7caa"} Sep 17 23:11:52.997656 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.997511 2559 scope.go:117] "RemoveContainer" containerID="19508f95523088e36a76440e545715e01f4ed404bedd591e7d1bc2827e649a35" Sep 17 23:11:52.999281 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.999256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" event={"ID":"8c9c9334-c345-4bfd-9977-814550d49937","Type":"ContainerStarted","Data":"05591ae5d8d549261ea31551cbd3848d5715d4ed5f1408f057eb937d2198525c"} Sep 17 23:11:52.999405 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:52.999317 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" Sep 17 23:11:53.013514 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:53.013456 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" podStartSLOduration=2.31472116 podStartE2EDuration="5.013443375s" totalImagesPullingTime="2.698722215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:53.012099637 +0000 UTC m=+341.129579111" watchObservedRunningTime="2026-09-17 23:11:53.013443375 +0000 UTC m=+341.130922859" Sep 17 23:11:53.026976 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:53.026955 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc"] Sep 17 23:11:53.032280 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:53.032256 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtckc"] Sep 17 23:11:53.047190 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:53.047136 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" podStartSLOduration=2.360974583 podStartE2EDuration="5.04712018s" totalImagesPullingTime="2.686145597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:53.046881332 +0000 UTC m=+341.164360802" watchObservedRunningTime="2026-09-17 23:11:53.04712018 +0000 UTC m=+341.164599628" Sep 17 23:11:54.286750 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:11:54.286722 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfce8216-c86e-40e5-93de-98bf1a83ea4a" path="/var/lib/kubelet/pods/bfce8216-c86e-40e5-93de-98bf1a83ea4a/volumes" Sep 17 23:12:24.008026 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:24.007994 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7ncm" Sep 17 23:12:24.011128 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:24.011103 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z7jjw" Sep 17 23:12:57.268679 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.268643 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp"] Sep 17 23:12:57.269115 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.269004 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfce8216-c86e-40e5-93de-98bf1a83ea4a" containerName="discovery" Sep 17 23:12:57.269115 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.269020 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfce8216-c86e-40e5-93de-98bf1a83ea4a" containerName="discovery" Sep 17 23:12:57.269115 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.269086 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfce8216-c86e-40e5-93de-98bf1a83ea4a" containerName="discovery" Sep 17 23:12:57.272009 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.271991 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp" Sep 17 23:12:57.275876 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.275855 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 17 23:12:57.276682 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.276663 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 17 23:12:57.276799 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.276681 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 17 23:12:57.276799 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.276700 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g67s6\"" Sep 17 23:12:57.287969 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.287943 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp"] Sep 17 23:12:57.362232 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.362193 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kml9f\" (UniqueName: \"kubernetes.io/projected/bc833510-c7aa-4a82-83d1-d55269d2ed31-kube-api-access-kml9f\") pod \"dns-operator-controller-manager-7b84cd7c9c-pvmfp\" (UID: \"bc833510-c7aa-4a82-83d1-d55269d2ed31\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp" Sep 17 23:12:57.462728 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.462685 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kml9f\" (UniqueName: \"kubernetes.io/projected/bc833510-c7aa-4a82-83d1-d55269d2ed31-kube-api-access-kml9f\") pod \"dns-operator-controller-manager-7b84cd7c9c-pvmfp\" (UID: \"bc833510-c7aa-4a82-83d1-d55269d2ed31\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp" Sep 17 23:12:57.477140 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.477111 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kml9f\" (UniqueName: \"kubernetes.io/projected/bc833510-c7aa-4a82-83d1-d55269d2ed31-kube-api-access-kml9f\") pod \"dns-operator-controller-manager-7b84cd7c9c-pvmfp\" (UID: \"bc833510-c7aa-4a82-83d1-d55269d2ed31\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp" Sep 17 23:12:57.581920 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.581806 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp" Sep 17 23:12:57.861471 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:57.861392 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp"] Sep 17 23:12:57.865230 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:12:57.865203 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc833510_c7aa_4a82_83d1_d55269d2ed31.slice/crio-c64d0527532b9c1103c366acdd0ee02ca4ec4502176c217e74af291936b9b8f2 WatchSource:0}: Error finding container c64d0527532b9c1103c366acdd0ee02ca4ec4502176c217e74af291936b9b8f2: Status 404 returned error can't find the container with id c64d0527532b9c1103c366acdd0ee02ca4ec4502176c217e74af291936b9b8f2 Sep 17 23:12:58.287322 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:58.287286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp" event={"ID":"bc833510-c7aa-4a82-83d1-d55269d2ed31","Type":"ContainerStarted","Data":"c64d0527532b9c1103c366acdd0ee02ca4ec4502176c217e74af291936b9b8f2"} Sep 17 23:12:59.488664 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:59.488623 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd"] Sep 17 23:12:59.493522 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:59.493500 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:12:59.496522 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:59.496500 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bfms9\"" Sep 17 23:12:59.503463 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:59.503438 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd"] Sep 17 23:12:59.583661 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:59.583631 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps77p\" (UniqueName: \"kubernetes.io/projected/497f7714-9e37-4278-87e5-6e234aff5edf-kube-api-access-ps77p\") pod \"limitador-operator-controller-manager-74495c69f8-8llhd\" (UID: \"497f7714-9e37-4278-87e5-6e234aff5edf\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:12:59.684537 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:59.684405 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps77p\" (UniqueName: \"kubernetes.io/projected/497f7714-9e37-4278-87e5-6e234aff5edf-kube-api-access-ps77p\") pod \"limitador-operator-controller-manager-74495c69f8-8llhd\" (UID: \"497f7714-9e37-4278-87e5-6e234aff5edf\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:12:59.694421 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:59.694362 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps77p\" (UniqueName: \"kubernetes.io/projected/497f7714-9e37-4278-87e5-6e234aff5edf-kube-api-access-ps77p\") pod \"limitador-operator-controller-manager-74495c69f8-8llhd\" (UID: \"497f7714-9e37-4278-87e5-6e234aff5edf\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:12:59.804614 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:12:59.804517 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:13:00.099847 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:00.099448 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd"] Sep 17 23:13:00.103272 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:13:00.103242 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497f7714_9e37_4278_87e5_6e234aff5edf.slice/crio-7c7ebec08eceb02475f9ae598f4546972a2d42a6faf6f022344d5471c86583bf WatchSource:0}: Error finding container 7c7ebec08eceb02475f9ae598f4546972a2d42a6faf6f022344d5471c86583bf: Status 404 returned error can't find the container with id 7c7ebec08eceb02475f9ae598f4546972a2d42a6faf6f022344d5471c86583bf Sep 17 23:13:00.298336 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:00.298299 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" event={"ID":"497f7714-9e37-4278-87e5-6e234aff5edf","Type":"ContainerStarted","Data":"7c7ebec08eceb02475f9ae598f4546972a2d42a6faf6f022344d5471c86583bf"} Sep 17 23:13:02.064865 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.064840 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-7jns7"] Sep 17 23:13:02.069549 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.069529 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-7jns7" Sep 17 23:13:02.071799 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.071781 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-svrc5\"" Sep 17 23:13:02.086380 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.086314 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-7jns7"] Sep 17 23:13:02.209410 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.209375 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgvmh\" (UniqueName: \"kubernetes.io/projected/a8c1ba28-60d8-42ef-a2fb-c5a3feddb06b-kube-api-access-cgvmh\") pod \"authorino-operator-76b478bd78-7jns7\" (UID: \"a8c1ba28-60d8-42ef-a2fb-c5a3feddb06b\") " pod="kuadrant-system/authorino-operator-76b478bd78-7jns7" Sep 17 23:13:02.308028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.307938 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp" event={"ID":"bc833510-c7aa-4a82-83d1-d55269d2ed31","Type":"ContainerStarted","Data":"548cb91659b02ec74fdaebdc1206f20f68ce9a82fbd273b10ea7ce2f18edee91"} Sep 17 23:13:02.308196 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.308099 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp" Sep 17 23:13:02.309534 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.309506 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" event={"ID":"497f7714-9e37-4278-87e5-6e234aff5edf","Type":"ContainerStarted","Data":"e09886d72bf11b5945d6757917b806e1ae602c00676eb60bdeddaf7e3fd4df6e"} Sep 17 23:13:02.309657 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.309642 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:13:02.309778 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.309762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgvmh\" (UniqueName: \"kubernetes.io/projected/a8c1ba28-60d8-42ef-a2fb-c5a3feddb06b-kube-api-access-cgvmh\") pod \"authorino-operator-76b478bd78-7jns7\" (UID: \"a8c1ba28-60d8-42ef-a2fb-c5a3feddb06b\") " pod="kuadrant-system/authorino-operator-76b478bd78-7jns7" Sep 17 23:13:02.317747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.317716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgvmh\" (UniqueName: \"kubernetes.io/projected/a8c1ba28-60d8-42ef-a2fb-c5a3feddb06b-kube-api-access-cgvmh\") pod \"authorino-operator-76b478bd78-7jns7\" (UID: \"a8c1ba28-60d8-42ef-a2fb-c5a3feddb06b\") " pod="kuadrant-system/authorino-operator-76b478bd78-7jns7" Sep 17 23:13:02.329359 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.329316 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp" podStartSLOduration=1.898927346 podStartE2EDuration="5.329301055s" totalImagesPullingTime="3.430373709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:02.328511193 +0000 UTC m=+410.445990667" watchObservedRunningTime="2026-09-17 23:13:02.329301055 +0000 UTC m=+410.446780505" Sep 17 23:13:02.345928 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.345861 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" podStartSLOduration=1.424536891 podStartE2EDuration="3.345845523s" totalImagesPullingTime="1.921308632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:02.343932034 +0000 UTC m=+410.461411498" watchObservedRunningTime="2026-09-17 23:13:02.345845523 +0000 UTC m=+410.463324973" Sep 17 23:13:02.379403 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.379370 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-7jns7" Sep 17 23:13:02.657390 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:02.657311 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-7jns7"] Sep 17 23:13:02.661203 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:13:02.661175 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c1ba28_60d8_42ef_a2fb_c5a3feddb06b.slice/crio-84608b622dfab654fff0e3c06891e6cc6599fae03d2e58f4be4279716326abec WatchSource:0}: Error finding container 84608b622dfab654fff0e3c06891e6cc6599fae03d2e58f4be4279716326abec: Status 404 returned error can't find the container with id 84608b622dfab654fff0e3c06891e6cc6599fae03d2e58f4be4279716326abec Sep 17 23:13:03.317286 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:03.317245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-7jns7" event={"ID":"a8c1ba28-60d8-42ef-a2fb-c5a3feddb06b","Type":"ContainerStarted","Data":"84608b622dfab654fff0e3c06891e6cc6599fae03d2e58f4be4279716326abec"} Sep 17 23:13:04.322663 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:04.322627 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-7jns7" event={"ID":"a8c1ba28-60d8-42ef-a2fb-c5a3feddb06b","Type":"ContainerStarted","Data":"5857ff676a94ca4375f261c1d1116f7e4305cef7b0f2f1cf7dfc489bad14611b"} Sep 17 23:13:04.323113 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:04.322698 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-7jns7" Sep 17 23:13:04.364858 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:04.364804 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-7jns7" podStartSLOduration=1.34507253 podStartE2EDuration="2.36478911s" totalImagesPullingTime="1.01971658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:04.363661621 +0000 UTC m=+412.481141077" watchObservedRunningTime="2026-09-17 23:13:04.36478911 +0000 UTC m=+412.482268560" Sep 17 23:13:06.485039 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:06.485007 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq"] Sep 17 23:13:06.488320 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:06.488302 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:06.491001 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:06.490972 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sjqbx\"" Sep 17 23:13:06.503519 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:06.503496 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq"] Sep 17 23:13:06.646129 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:06.646076 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb1ea228-b312-49d5-b8a6-9cbb03629bfe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" (UID: \"fb1ea228-b312-49d5-b8a6-9cbb03629bfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:06.646318 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:06.646190 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92rz6\" (UniqueName: \"kubernetes.io/projected/fb1ea228-b312-49d5-b8a6-9cbb03629bfe-kube-api-access-92rz6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" (UID: \"fb1ea228-b312-49d5-b8a6-9cbb03629bfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:06.747310 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:06.747218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92rz6\" (UniqueName: \"kubernetes.io/projected/fb1ea228-b312-49d5-b8a6-9cbb03629bfe-kube-api-access-92rz6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" (UID: \"fb1ea228-b312-49d5-b8a6-9cbb03629bfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:06.747310 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:06.747291 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb1ea228-b312-49d5-b8a6-9cbb03629bfe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" (UID: \"fb1ea228-b312-49d5-b8a6-9cbb03629bfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:06.747688 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:06.747667 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb1ea228-b312-49d5-b8a6-9cbb03629bfe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" (UID: \"fb1ea228-b312-49d5-b8a6-9cbb03629bfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:06.758077 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:06.758052 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92rz6\" (UniqueName: \"kubernetes.io/projected/fb1ea228-b312-49d5-b8a6-9cbb03629bfe-kube-api-access-92rz6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" (UID: \"fb1ea228-b312-49d5-b8a6-9cbb03629bfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:06.798993 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:06.798963 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:07.087867 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:07.087837 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq"] Sep 17 23:13:07.090401 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:13:07.090372 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1ea228_b312_49d5_b8a6_9cbb03629bfe.slice/crio-1b2c079dfe29499449d47b4ef5087bfbad416da16fefd5d44760a5da1a7e8705 WatchSource:0}: Error finding container 1b2c079dfe29499449d47b4ef5087bfbad416da16fefd5d44760a5da1a7e8705: Status 404 returned error can't find the container with id 1b2c079dfe29499449d47b4ef5087bfbad416da16fefd5d44760a5da1a7e8705 Sep 17 23:13:07.337780 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:07.337693 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" event={"ID":"fb1ea228-b312-49d5-b8a6-9cbb03629bfe","Type":"ContainerStarted","Data":"1b2c079dfe29499449d47b4ef5087bfbad416da16fefd5d44760a5da1a7e8705"} Sep 17 23:13:11.360257 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:11.360217 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" event={"ID":"fb1ea228-b312-49d5-b8a6-9cbb03629bfe","Type":"ContainerStarted","Data":"974c2325a9da75c5d146b22812593ba05ebdc61f7c589b6a2269a13ca825614c"} Sep 17 23:13:11.360626 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:11.360356 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:11.391083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:11.391010 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" podStartSLOduration=1.752447498 podStartE2EDuration="5.390995334s" totalImagesPullingTime="3.638547836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:11.38804661 +0000 UTC m=+419.505526077" watchObservedRunningTime="2026-09-17 23:13:11.390995334 +0000 UTC m=+419.508474783" Sep 17 23:13:13.319369 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:13.319339 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:13:13.319764 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:13.319412 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pvmfp" Sep 17 23:13:15.330162 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:15.330132 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-7jns7" Sep 17 23:13:22.367007 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:22.366975 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:24.557347 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.557310 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb"] Sep 17 23:13:24.563431 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.563411 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq"] Sep 17 23:13:24.563567 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.563549 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" podUID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" containerName="manager" containerID="cri-o://974c2325a9da75c5d146b22812593ba05ebdc61f7c589b6a2269a13ca825614c" gracePeriod=2 Sep 17 23:13:24.563614 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.563589 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" Sep 17 23:13:24.569588 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.569562 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq"] Sep 17 23:13:24.577547 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.577524 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb"] Sep 17 23:13:24.577713 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:24.577567 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:24.582602 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.582577 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd"] Sep 17 23:13:24.582849 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.582797 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" podUID="497f7714-9e37-4278-87e5-6e234aff5edf" containerName="manager" containerID="cri-o://e09886d72bf11b5945d6757917b806e1ae602c00676eb60bdeddaf7e3fd4df6e" gracePeriod=2 Sep 17 23:13:24.589334 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.589306 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-kbllb\" (UID: \"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" Sep 17 23:13:24.589456 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.589413 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z977r\" (UniqueName: \"kubernetes.io/projected/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc-kube-api-access-z977r\") pod \"kuadrant-operator-controller-manager-5f685c496f-kbllb\" (UID: \"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" Sep 17 23:13:24.589869 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.589850 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9"] Sep 17 23:13:24.590234 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.590217 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" containerName="manager" Sep 17 23:13:24.590303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.590237 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" containerName="manager" Sep 17 23:13:24.590369 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.590356 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" containerName="manager" Sep 17 23:13:24.594477 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.594457 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd"] Sep 17 23:13:24.594559 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.594539 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" Sep 17 23:13:24.605565 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:24.605539 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8llhd\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="497f7714-9e37-4278-87e5-6e234aff5edf" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:13:24.606959 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.606935 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5"] Sep 17 23:13:24.607391 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.607371 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497f7714-9e37-4278-87e5-6e234aff5edf" containerName="manager" Sep 17 23:13:24.607391 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.607393 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="497f7714-9e37-4278-87e5-6e234aff5edf" containerName="manager" Sep 17 23:13:24.607532 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.607507 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="497f7714-9e37-4278-87e5-6e234aff5edf" containerName="manager" Sep 17 23:13:24.608119 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:24.608091 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:24.610924 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.610888 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9"] Sep 17 23:13:24.611082 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.611060 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5" Sep 17 23:13:24.627555 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.627530 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5"] Sep 17 23:13:24.641751 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:24.641715 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8llhd\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="497f7714-9e37-4278-87e5-6e234aff5edf" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:13:24.664213 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:24.664178 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:24.689857 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.689782 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6lk\" (UniqueName: \"kubernetes.io/projected/e71e542a-851c-4d11-a715-bcb7cf13b6c1-kube-api-access-2n6lk\") pod \"limitador-operator-controller-manager-74495c69f8-v6tp5\" (UID: \"e71e542a-851c-4d11-a715-bcb7cf13b6c1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5" Sep 17 23:13:24.689857 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.689816 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z977r\" (UniqueName: \"kubernetes.io/projected/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc-kube-api-access-z977r\") pod \"kuadrant-operator-controller-manager-5f685c496f-kbllb\" (UID: \"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" Sep 17 23:13:24.690023 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.689916 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-kbllb\" (UID: \"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" Sep 17 23:13:24.690023 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.689959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd4c4\" (UniqueName: \"kubernetes.io/projected/313f8bf4-4805-4b57-b743-9495d1ae5ed3-kube-api-access-qd4c4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4wjc9\" (UID: \"313f8bf4-4805-4b57-b743-9495d1ae5ed3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" Sep 17 23:13:24.690023 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.689988 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/313f8bf4-4805-4b57-b743-9495d1ae5ed3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4wjc9\" (UID: \"313f8bf4-4805-4b57-b743-9495d1ae5ed3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" Sep 17 23:13:24.690313 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.690290 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-kbllb\" (UID: \"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" Sep 17 23:13:24.698279 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.698257 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z977r\" (UniqueName: \"kubernetes.io/projected/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc-kube-api-access-z977r\") pod \"kuadrant-operator-controller-manager-5f685c496f-kbllb\" (UID: \"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" Sep 17 23:13:24.790423 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.790391 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd4c4\" (UniqueName: \"kubernetes.io/projected/313f8bf4-4805-4b57-b743-9495d1ae5ed3-kube-api-access-qd4c4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4wjc9\" (UID: \"313f8bf4-4805-4b57-b743-9495d1ae5ed3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" Sep 17 23:13:24.790423 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.790428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/313f8bf4-4805-4b57-b743-9495d1ae5ed3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4wjc9\" (UID: \"313f8bf4-4805-4b57-b743-9495d1ae5ed3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" Sep 17 23:13:24.790604 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.790469 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6lk\" (UniqueName: \"kubernetes.io/projected/e71e542a-851c-4d11-a715-bcb7cf13b6c1-kube-api-access-2n6lk\") pod \"limitador-operator-controller-manager-74495c69f8-v6tp5\" (UID: \"e71e542a-851c-4d11-a715-bcb7cf13b6c1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5" Sep 17 23:13:24.790836 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.790818 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/313f8bf4-4805-4b57-b743-9495d1ae5ed3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4wjc9\" (UID: \"313f8bf4-4805-4b57-b743-9495d1ae5ed3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" Sep 17 23:13:24.802386 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.802358 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6lk\" (UniqueName: \"kubernetes.io/projected/e71e542a-851c-4d11-a715-bcb7cf13b6c1-kube-api-access-2n6lk\") pod \"limitador-operator-controller-manager-74495c69f8-v6tp5\" (UID: \"e71e542a-851c-4d11-a715-bcb7cf13b6c1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5" Sep 17 23:13:24.802739 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.802719 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd4c4\" (UniqueName: \"kubernetes.io/projected/313f8bf4-4805-4b57-b743-9495d1ae5ed3-kube-api-access-qd4c4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4wjc9\" (UID: \"313f8bf4-4805-4b57-b743-9495d1ae5ed3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" Sep 17 23:13:24.815446 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.815426 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:24.817644 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:24.817620 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8llhd\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="497f7714-9e37-4278-87e5-6e234aff5edf" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:13:24.817743 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.817728 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:13:24.819878 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:24.819857 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:24.824092 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:24.824068 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:24.826011 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:24.825990 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8llhd\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="497f7714-9e37-4278-87e5-6e234aff5edf" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:13:24.875204 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.875177 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" Sep 17 23:13:24.891211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.891185 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92rz6\" (UniqueName: \"kubernetes.io/projected/fb1ea228-b312-49d5-b8a6-9cbb03629bfe-kube-api-access-92rz6\") pod \"fb1ea228-b312-49d5-b8a6-9cbb03629bfe\" (UID: \"fb1ea228-b312-49d5-b8a6-9cbb03629bfe\") " Sep 17 23:13:24.891296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.891237 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps77p\" (UniqueName: \"kubernetes.io/projected/497f7714-9e37-4278-87e5-6e234aff5edf-kube-api-access-ps77p\") pod \"497f7714-9e37-4278-87e5-6e234aff5edf\" (UID: \"497f7714-9e37-4278-87e5-6e234aff5edf\") " Sep 17 23:13:24.891352 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.891322 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb1ea228-b312-49d5-b8a6-9cbb03629bfe-extensions-socket-volume\") pod \"fb1ea228-b312-49d5-b8a6-9cbb03629bfe\" (UID: \"fb1ea228-b312-49d5-b8a6-9cbb03629bfe\") " Sep 17 23:13:24.891681 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.891653 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1ea228-b312-49d5-b8a6-9cbb03629bfe-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fb1ea228-b312-49d5-b8a6-9cbb03629bfe" (UID: "fb1ea228-b312-49d5-b8a6-9cbb03629bfe"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:13:24.893402 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.893375 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497f7714-9e37-4278-87e5-6e234aff5edf-kube-api-access-ps77p" (OuterVolumeSpecName: "kube-api-access-ps77p") pod "497f7714-9e37-4278-87e5-6e234aff5edf" (UID: "497f7714-9e37-4278-87e5-6e234aff5edf"). InnerVolumeSpecName "kube-api-access-ps77p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:13:24.893482 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.893393 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1ea228-b312-49d5-b8a6-9cbb03629bfe-kube-api-access-92rz6" (OuterVolumeSpecName: "kube-api-access-92rz6") pod "fb1ea228-b312-49d5-b8a6-9cbb03629bfe" (UID: "fb1ea228-b312-49d5-b8a6-9cbb03629bfe"). InnerVolumeSpecName "kube-api-access-92rz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:13:24.998347 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.998254 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92rz6\" (UniqueName: \"kubernetes.io/projected/fb1ea228-b312-49d5-b8a6-9cbb03629bfe-kube-api-access-92rz6\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:13:24.998523 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.998442 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps77p\" (UniqueName: \"kubernetes.io/projected/497f7714-9e37-4278-87e5-6e234aff5edf-kube-api-access-ps77p\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:13:24.998523 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:24.998461 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb1ea228-b312-49d5-b8a6-9cbb03629bfe-extensions-socket-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:13:25.015876 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.015847 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" Sep 17 23:13:25.023274 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.023245 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5" Sep 17 23:13:25.153697 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.153642 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb"] Sep 17 23:13:25.156191 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:13:25.156168 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64fc11e_4eff_42ee_8c9a_77a5649ef7cc.slice/crio-2d7881e7f57baf9dd0c38093263514deb47e5a274987d0158559614f0ee2753e WatchSource:0}: Error finding container 2d7881e7f57baf9dd0c38093263514deb47e5a274987d0158559614f0ee2753e: Status 404 returned error can't find the container with id 2d7881e7f57baf9dd0c38093263514deb47e5a274987d0158559614f0ee2753e Sep 17 23:13:25.298580 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.298463 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9"] Sep 17 23:13:25.301421 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:13:25.301396 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313f8bf4_4805_4b57_b743_9495d1ae5ed3.slice/crio-3de62ef918f5cc57b59cf731a81fb8141a8c401f56ead92db2b437866e2756be WatchSource:0}: Error finding container 3de62ef918f5cc57b59cf731a81fb8141a8c401f56ead92db2b437866e2756be: Status 404 returned error can't find the container with id 3de62ef918f5cc57b59cf731a81fb8141a8c401f56ead92db2b437866e2756be Sep 17 23:13:25.310040 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.310016 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5"] Sep 17 23:13:25.313069 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:13:25.313045 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71e542a_851c_4d11_a715_bcb7cf13b6c1.slice/crio-57ddf69a253831d42c3e6151c9e7baeef256804c48f2db2a929e528ac6770d66 WatchSource:0}: Error finding container 57ddf69a253831d42c3e6151c9e7baeef256804c48f2db2a929e528ac6770d66: Status 404 returned error can't find the container with id 57ddf69a253831d42c3e6151c9e7baeef256804c48f2db2a929e528ac6770d66 Sep 17 23:13:25.428372 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.428331 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5" event={"ID":"e71e542a-851c-4d11-a715-bcb7cf13b6c1","Type":"ContainerStarted","Data":"a1d080ff8fad44b7000eae28277058dec5dfbf6673a79b741e30edd9df0829bf"} Sep 17 23:13:25.428570 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.428473 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5" Sep 17 23:13:25.428570 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.428512 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5" event={"ID":"e71e542a-851c-4d11-a715-bcb7cf13b6c1","Type":"ContainerStarted","Data":"57ddf69a253831d42c3e6151c9e7baeef256804c48f2db2a929e528ac6770d66"} Sep 17 23:13:25.429971 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.429947 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" event={"ID":"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc","Type":"ContainerStarted","Data":"4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54"} Sep 17 23:13:25.430086 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.429984 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" event={"ID":"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc","Type":"ContainerStarted","Data":"2d7881e7f57baf9dd0c38093263514deb47e5a274987d0158559614f0ee2753e"} Sep 17 23:13:25.430086 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.430041 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" Sep 17 23:13:25.430377 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:25.430351 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8llhd\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="497f7714-9e37-4278-87e5-6e234aff5edf" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:13:25.431641 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.431618 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" event={"ID":"313f8bf4-4805-4b57-b743-9495d1ae5ed3","Type":"ContainerStarted","Data":"b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e"} Sep 17 23:13:25.431739 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.431657 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" event={"ID":"313f8bf4-4805-4b57-b743-9495d1ae5ed3","Type":"ContainerStarted","Data":"3de62ef918f5cc57b59cf731a81fb8141a8c401f56ead92db2b437866e2756be"} Sep 17 23:13:25.431739 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.431684 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" Sep 17 23:13:25.432825 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.432807 2559 generic.go:356] "Generic (PLEG): container finished" podID="497f7714-9e37-4278-87e5-6e234aff5edf" containerID="e09886d72bf11b5945d6757917b806e1ae602c00676eb60bdeddaf7e3fd4df6e" exitCode=0 Sep 17 23:13:25.432933 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.432858 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:13:25.432933 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.432925 2559 scope.go:117] "RemoveContainer" containerID="e09886d72bf11b5945d6757917b806e1ae602c00676eb60bdeddaf7e3fd4df6e" Sep 17 23:13:25.434381 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.434358 2559 generic.go:356] "Generic (PLEG): container finished" podID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" containerID="974c2325a9da75c5d146b22812593ba05ebdc61f7c589b6a2269a13ca825614c" exitCode=0 Sep 17 23:13:25.434489 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.434428 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:25.442513 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.442492 2559 scope.go:117] "RemoveContainer" containerID="e09886d72bf11b5945d6757917b806e1ae602c00676eb60bdeddaf7e3fd4df6e" Sep 17 23:13:25.442867 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:25.442839 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09886d72bf11b5945d6757917b806e1ae602c00676eb60bdeddaf7e3fd4df6e\": container with ID starting with e09886d72bf11b5945d6757917b806e1ae602c00676eb60bdeddaf7e3fd4df6e not found: ID does not exist" containerID="e09886d72bf11b5945d6757917b806e1ae602c00676eb60bdeddaf7e3fd4df6e" Sep 17 23:13:25.443018 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.442879 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09886d72bf11b5945d6757917b806e1ae602c00676eb60bdeddaf7e3fd4df6e"} err="failed to get container status \"e09886d72bf11b5945d6757917b806e1ae602c00676eb60bdeddaf7e3fd4df6e\": rpc error: code = NotFound desc = could not find container \"e09886d72bf11b5945d6757917b806e1ae602c00676eb60bdeddaf7e3fd4df6e\": container with ID starting with e09886d72bf11b5945d6757917b806e1ae602c00676eb60bdeddaf7e3fd4df6e not found: ID does not exist" Sep 17 23:13:25.443018 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.442954 2559 scope.go:117] "RemoveContainer" containerID="974c2325a9da75c5d146b22812593ba05ebdc61f7c589b6a2269a13ca825614c" Sep 17 23:13:25.453488 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:25.453380 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:25.453566 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.453511 2559 scope.go:117] "RemoveContainer" containerID="974c2325a9da75c5d146b22812593ba05ebdc61f7c589b6a2269a13ca825614c" Sep 17 23:13:25.453839 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:25.453816 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974c2325a9da75c5d146b22812593ba05ebdc61f7c589b6a2269a13ca825614c\": container with ID starting with 974c2325a9da75c5d146b22812593ba05ebdc61f7c589b6a2269a13ca825614c not found: ID does not exist" containerID="974c2325a9da75c5d146b22812593ba05ebdc61f7c589b6a2269a13ca825614c" Sep 17 23:13:25.453970 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.453847 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974c2325a9da75c5d146b22812593ba05ebdc61f7c589b6a2269a13ca825614c"} err="failed to get container status \"974c2325a9da75c5d146b22812593ba05ebdc61f7c589b6a2269a13ca825614c\": rpc error: code = NotFound desc = could not find container \"974c2325a9da75c5d146b22812593ba05ebdc61f7c589b6a2269a13ca825614c\": container with ID starting with 974c2325a9da75c5d146b22812593ba05ebdc61f7c589b6a2269a13ca825614c not found: ID does not exist" Sep 17 23:13:25.473983 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.473928 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5" podStartSLOduration=1.473887882 podStartE2EDuration="1.473887882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:25.45150833 +0000 UTC m=+433.568987803" watchObservedRunningTime="2026-09-17 23:13:25.473887882 +0000 UTC m=+433.591367330" Sep 17 23:13:25.475485 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.475452 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" podStartSLOduration=1.475442162 podStartE2EDuration="1.475442162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:25.473731631 +0000 UTC m=+433.591211100" watchObservedRunningTime="2026-09-17 23:13:25.475442162 +0000 UTC m=+433.592921612" Sep 17 23:13:25.475727 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:25.475696 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8llhd\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="497f7714-9e37-4278-87e5-6e234aff5edf" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8llhd" Sep 17 23:13:25.498792 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:25.498746 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" podStartSLOduration=1.498726997 podStartE2EDuration="1.498726997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:25.497951034 +0000 UTC m=+433.615430483" watchObservedRunningTime="2026-09-17 23:13:25.498726997 +0000 UTC m=+433.616206446" Sep 17 23:13:25.500330 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:25.500305 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jqrmq\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jqrmq" Sep 17 23:13:26.287190 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:26.287156 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="497f7714-9e37-4278-87e5-6e234aff5edf" path="/var/lib/kubelet/pods/497f7714-9e37-4278-87e5-6e234aff5edf/volumes" Sep 17 23:13:26.287559 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:26.287506 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb1ea228-b312-49d5-b8a6-9cbb03629bfe" path="/var/lib/kubelet/pods/fb1ea228-b312-49d5-b8a6-9cbb03629bfe/volumes" Sep 17 23:13:36.443998 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.443969 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v6tp5" Sep 17 23:13:36.444469 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.444031 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" Sep 17 23:13:36.444469 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.444073 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" Sep 17 23:13:36.524661 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.524627 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9"] Sep 17 23:13:36.524845 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.524827 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" podUID="313f8bf4-4805-4b57-b743-9495d1ae5ed3" containerName="manager" containerID="cri-o://b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e" gracePeriod=10 Sep 17 23:13:36.762052 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.762028 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" Sep 17 23:13:36.803413 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.803385 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd4c4\" (UniqueName: \"kubernetes.io/projected/313f8bf4-4805-4b57-b743-9495d1ae5ed3-kube-api-access-qd4c4\") pod \"313f8bf4-4805-4b57-b743-9495d1ae5ed3\" (UID: \"313f8bf4-4805-4b57-b743-9495d1ae5ed3\") " Sep 17 23:13:36.803581 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.803488 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/313f8bf4-4805-4b57-b743-9495d1ae5ed3-extensions-socket-volume\") pod \"313f8bf4-4805-4b57-b743-9495d1ae5ed3\" (UID: \"313f8bf4-4805-4b57-b743-9495d1ae5ed3\") " Sep 17 23:13:36.803870 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.803846 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313f8bf4-4805-4b57-b743-9495d1ae5ed3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "313f8bf4-4805-4b57-b743-9495d1ae5ed3" (UID: "313f8bf4-4805-4b57-b743-9495d1ae5ed3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:13:36.805545 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.805525 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313f8bf4-4805-4b57-b743-9495d1ae5ed3-kube-api-access-qd4c4" (OuterVolumeSpecName: "kube-api-access-qd4c4") pod "313f8bf4-4805-4b57-b743-9495d1ae5ed3" (UID: "313f8bf4-4805-4b57-b743-9495d1ae5ed3"). InnerVolumeSpecName "kube-api-access-qd4c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:13:36.904750 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.904718 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/313f8bf4-4805-4b57-b743-9495d1ae5ed3-extensions-socket-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:13:36.904750 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.904745 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd4c4\" (UniqueName: \"kubernetes.io/projected/313f8bf4-4805-4b57-b743-9495d1ae5ed3-kube-api-access-qd4c4\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:13:36.931220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.931187 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh"] Sep 17 23:13:36.931618 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.931601 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="313f8bf4-4805-4b57-b743-9495d1ae5ed3" containerName="manager" Sep 17 23:13:36.931695 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.931620 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="313f8bf4-4805-4b57-b743-9495d1ae5ed3" containerName="manager" Sep 17 23:13:36.931746 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.931705 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="313f8bf4-4805-4b57-b743-9495d1ae5ed3" containerName="manager" Sep 17 23:13:36.933714 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.933695 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:13:36.951910 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:36.951874 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh"] Sep 17 23:13:37.005988 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.005923 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbl7\" (UniqueName: \"kubernetes.io/projected/572db407-342d-4115-aa58-fe567fc83eb9-kube-api-access-gkbl7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-brvgh\" (UID: \"572db407-342d-4115-aa58-fe567fc83eb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:13:37.005988 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.005976 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/572db407-342d-4115-aa58-fe567fc83eb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-brvgh\" (UID: \"572db407-342d-4115-aa58-fe567fc83eb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:13:37.107352 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.107325 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbl7\" (UniqueName: \"kubernetes.io/projected/572db407-342d-4115-aa58-fe567fc83eb9-kube-api-access-gkbl7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-brvgh\" (UID: \"572db407-342d-4115-aa58-fe567fc83eb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:13:37.107461 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.107359 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/572db407-342d-4115-aa58-fe567fc83eb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-brvgh\" (UID: \"572db407-342d-4115-aa58-fe567fc83eb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:13:37.107657 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.107643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/572db407-342d-4115-aa58-fe567fc83eb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-brvgh\" (UID: \"572db407-342d-4115-aa58-fe567fc83eb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:13:37.120946 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.120924 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbl7\" (UniqueName: \"kubernetes.io/projected/572db407-342d-4115-aa58-fe567fc83eb9-kube-api-access-gkbl7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-brvgh\" (UID: \"572db407-342d-4115-aa58-fe567fc83eb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:13:37.242706 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.242681 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:13:37.504304 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.504274 2559 generic.go:356] "Generic (PLEG): container finished" podID="313f8bf4-4805-4b57-b743-9495d1ae5ed3" containerID="b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e" exitCode=0 Sep 17 23:13:37.504697 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.504315 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" Sep 17 23:13:37.504697 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.504320 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" event={"ID":"313f8bf4-4805-4b57-b743-9495d1ae5ed3","Type":"ContainerDied","Data":"b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e"} Sep 17 23:13:37.504697 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.504443 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9" event={"ID":"313f8bf4-4805-4b57-b743-9495d1ae5ed3","Type":"ContainerDied","Data":"3de62ef918f5cc57b59cf731a81fb8141a8c401f56ead92db2b437866e2756be"} Sep 17 23:13:37.504697 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.504473 2559 scope.go:117] "RemoveContainer" containerID="b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e" Sep 17 23:13:37.513285 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.513263 2559 scope.go:117] "RemoveContainer" containerID="b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e" Sep 17 23:13:37.513729 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:37.513709 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e\": container with ID starting with b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e not found: ID does not exist" containerID="b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e" Sep 17 23:13:37.513818 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.513738 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e"} err="failed to get container status \"b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e\": rpc error: code = NotFound desc = could not find container \"b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e\": container with ID starting with b828326fcc7012e8ef07dba3168b4be5ebd566e266b76e66945eb5e6dda6fe2e not found: ID does not exist" Sep 17 23:13:37.516534 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.516515 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh"] Sep 17 23:13:37.518356 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:13:37.518339 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572db407_342d_4115_aa58_fe567fc83eb9.slice/crio-b5b83a8cda955e67ff67755b8450fb8370c7234f643f00cbd5fa95dfe0ba2f09 WatchSource:0}: Error finding container b5b83a8cda955e67ff67755b8450fb8370c7234f643f00cbd5fa95dfe0ba2f09: Status 404 returned error can't find the container with id b5b83a8cda955e67ff67755b8450fb8370c7234f643f00cbd5fa95dfe0ba2f09 Sep 17 23:13:37.530459 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.530438 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9"] Sep 17 23:13:37.538872 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:37.538824 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4wjc9"] Sep 17 23:13:38.287164 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:38.287131 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="313f8bf4-4805-4b57-b743-9495d1ae5ed3" path="/var/lib/kubelet/pods/313f8bf4-4805-4b57-b743-9495d1ae5ed3/volumes" Sep 17 23:13:38.511604 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:38.511567 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" event={"ID":"572db407-342d-4115-aa58-fe567fc83eb9","Type":"ContainerStarted","Data":"6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483"} Sep 17 23:13:38.512061 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:38.511633 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" event={"ID":"572db407-342d-4115-aa58-fe567fc83eb9","Type":"ContainerStarted","Data":"b5b83a8cda955e67ff67755b8450fb8370c7234f643f00cbd5fa95dfe0ba2f09"} Sep 17 23:13:38.512061 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:38.511764 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:13:38.537484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:38.537405 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" podStartSLOduration=2.537390937 podStartE2EDuration="2.537390937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:38.534648894 +0000 UTC m=+446.652128363" watchObservedRunningTime="2026-09-17 23:13:38.537390937 +0000 UTC m=+446.654870387" Sep 17 23:13:49.520229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:49.520199 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:13:49.573708 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:49.573678 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb"] Sep 17 23:13:49.573933 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:49.573895 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" podUID="a64fc11e-4eff-42ee-8c9a-77a5649ef7cc" containerName="manager" containerID="cri-o://4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54" gracePeriod=10 Sep 17 23:13:49.811582 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:49.811558 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" Sep 17 23:13:49.909265 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:49.909233 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc-extensions-socket-volume\") pod \"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc\" (UID: \"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc\") " Sep 17 23:13:49.909423 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:49.909298 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z977r\" (UniqueName: \"kubernetes.io/projected/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc-kube-api-access-z977r\") pod \"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc\" (UID: \"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc\") " Sep 17 23:13:49.909602 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:49.909581 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a64fc11e-4eff-42ee-8c9a-77a5649ef7cc" (UID: "a64fc11e-4eff-42ee-8c9a-77a5649ef7cc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:13:49.911344 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:49.911316 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc-kube-api-access-z977r" (OuterVolumeSpecName: "kube-api-access-z977r") pod "a64fc11e-4eff-42ee-8c9a-77a5649ef7cc" (UID: "a64fc11e-4eff-42ee-8c9a-77a5649ef7cc"). InnerVolumeSpecName "kube-api-access-z977r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:13:50.010392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:50.010368 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc-extensions-socket-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:13:50.010392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:50.010388 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z977r\" (UniqueName: \"kubernetes.io/projected/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc-kube-api-access-z977r\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:13:50.571054 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:50.570977 2559 generic.go:356] "Generic (PLEG): container finished" podID="a64fc11e-4eff-42ee-8c9a-77a5649ef7cc" containerID="4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54" exitCode=0 Sep 17 23:13:50.571054 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:50.571035 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" Sep 17 23:13:50.571472 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:50.571048 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" event={"ID":"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc","Type":"ContainerDied","Data":"4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54"} Sep 17 23:13:50.571472 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:50.571097 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb" event={"ID":"a64fc11e-4eff-42ee-8c9a-77a5649ef7cc","Type":"ContainerDied","Data":"2d7881e7f57baf9dd0c38093263514deb47e5a274987d0158559614f0ee2753e"} Sep 17 23:13:50.571472 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:50.571118 2559 scope.go:117] "RemoveContainer" containerID="4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54" Sep 17 23:13:50.579334 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:50.579156 2559 scope.go:117] "RemoveContainer" containerID="4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54" Sep 17 23:13:50.579460 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:13:50.579441 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54\": container with ID starting with 4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54 not found: ID does not exist" containerID="4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54" Sep 17 23:13:50.579502 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:50.579470 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54"} err="failed to get container status \"4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54\": rpc error: code = NotFound desc = could not find container \"4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54\": container with ID starting with 4ee7d7398192dac59a939521ac3bf116ef47a5c983423d02638a03564cdaad54 not found: ID does not exist" Sep 17 23:13:50.590725 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:50.590704 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb"] Sep 17 23:13:50.594827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:50.594808 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kbllb"] Sep 17 23:13:52.287619 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:13:52.287587 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a64fc11e-4eff-42ee-8c9a-77a5649ef7cc" path="/var/lib/kubelet/pods/a64fc11e-4eff-42ee-8c9a-77a5649ef7cc/volumes" Sep 17 23:14:07.971117 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:07.971080 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:14:07.971576 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:07.971423 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a64fc11e-4eff-42ee-8c9a-77a5649ef7cc" containerName="manager" Sep 17 23:14:07.971576 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:07.971435 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a64fc11e-4eff-42ee-8c9a-77a5649ef7cc" containerName="manager" Sep 17 23:14:07.971576 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:07.971498 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a64fc11e-4eff-42ee-8c9a-77a5649ef7cc" containerName="manager" Sep 17 23:14:07.974727 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:07.974705 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" Sep 17 23:14:07.977026 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:07.977000 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 17 23:14:07.977139 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:07.977047 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cqh4k\"" Sep 17 23:14:07.985490 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:07.985466 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:14:08.002918 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:07.997465 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:14:08.150243 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:08.150207 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e570d1d6-9e91-4e60-90ea-74d24497c1d8-config-file\") pod \"limitador-limitador-84994f7ffc-rn8c6\" (UID: \"e570d1d6-9e91-4e60-90ea-74d24497c1d8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" Sep 17 23:14:08.150412 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:08.150281 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbb8r\" (UniqueName: \"kubernetes.io/projected/e570d1d6-9e91-4e60-90ea-74d24497c1d8-kube-api-access-mbb8r\") pod \"limitador-limitador-84994f7ffc-rn8c6\" (UID: \"e570d1d6-9e91-4e60-90ea-74d24497c1d8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" Sep 17 23:14:08.251595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:08.251528 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbb8r\" (UniqueName: \"kubernetes.io/projected/e570d1d6-9e91-4e60-90ea-74d24497c1d8-kube-api-access-mbb8r\") pod \"limitador-limitador-84994f7ffc-rn8c6\" (UID: \"e570d1d6-9e91-4e60-90ea-74d24497c1d8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" Sep 17 23:14:08.251595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:08.251573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e570d1d6-9e91-4e60-90ea-74d24497c1d8-config-file\") pod \"limitador-limitador-84994f7ffc-rn8c6\" (UID: \"e570d1d6-9e91-4e60-90ea-74d24497c1d8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" Sep 17 23:14:08.252216 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:08.252197 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e570d1d6-9e91-4e60-90ea-74d24497c1d8-config-file\") pod \"limitador-limitador-84994f7ffc-rn8c6\" (UID: \"e570d1d6-9e91-4e60-90ea-74d24497c1d8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" Sep 17 23:14:08.259214 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:08.259191 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbb8r\" (UniqueName: \"kubernetes.io/projected/e570d1d6-9e91-4e60-90ea-74d24497c1d8-kube-api-access-mbb8r\") pod \"limitador-limitador-84994f7ffc-rn8c6\" (UID: \"e570d1d6-9e91-4e60-90ea-74d24497c1d8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" Sep 17 23:14:08.284892 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:08.284868 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" Sep 17 23:14:08.557475 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:08.557283 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:14:08.560688 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:14:08.560657 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode570d1d6_9e91_4e60_90ea_74d24497c1d8.slice/crio-e00c6ae4b03d110836edb865b194d187cb6dfbd2ca54eb7638d0ebe190b2e08f WatchSource:0}: Error finding container e00c6ae4b03d110836edb865b194d187cb6dfbd2ca54eb7638d0ebe190b2e08f: Status 404 returned error can't find the container with id e00c6ae4b03d110836edb865b194d187cb6dfbd2ca54eb7638d0ebe190b2e08f Sep 17 23:14:08.663312 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:08.663276 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" event={"ID":"e570d1d6-9e91-4e60-90ea-74d24497c1d8","Type":"ContainerStarted","Data":"e00c6ae4b03d110836edb865b194d187cb6dfbd2ca54eb7638d0ebe190b2e08f"} Sep 17 23:14:09.668803 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:09.668774 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" event={"ID":"e570d1d6-9e91-4e60-90ea-74d24497c1d8","Type":"ContainerStarted","Data":"cddb6f12d2585eada7efe9014002d0be759a50c5dc0831c8c139951567260ec4"} Sep 17 23:14:09.669208 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:09.668841 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" Sep 17 23:14:09.685316 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:09.685273 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" podStartSLOduration=1.712795616 podStartE2EDuration="2.685256938s" totalImagesPullingTime="972.461322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:09.684088277 +0000 UTC m=+477.801567741" watchObservedRunningTime="2026-09-17 23:14:09.685256938 +0000 UTC m=+477.802736387" Sep 17 23:14:20.675125 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:14:20.675090 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-rn8c6" Sep 17 23:15:21.405983 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.405950 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-x5r9q"] Sep 17 23:15:21.409743 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.409721 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" Sep 17 23:15:21.412294 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.412274 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 17 23:15:21.412400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.412312 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xbsm5\"" Sep 17 23:15:21.420500 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.420467 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-x5r9q"] Sep 17 23:15:21.477237 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.477205 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:15:21.506065 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.506037 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2gt\" (UniqueName: \"kubernetes.io/projected/c2f4d315-2130-4dc1-8c78-fca47740b436-kube-api-access-7t2gt\") pod \"payload-processing-5b54b7fbb7-x5r9q\" (UID: \"c2f4d315-2130-4dc1-8c78-fca47740b436\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" Sep 17 23:15:21.506210 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.506096 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f4d315-2130-4dc1-8c78-fca47740b436-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-x5r9q\" (UID: \"c2f4d315-2130-4dc1-8c78-fca47740b436\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" Sep 17 23:15:21.607117 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.607087 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2gt\" (UniqueName: \"kubernetes.io/projected/c2f4d315-2130-4dc1-8c78-fca47740b436-kube-api-access-7t2gt\") pod \"payload-processing-5b54b7fbb7-x5r9q\" (UID: \"c2f4d315-2130-4dc1-8c78-fca47740b436\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" Sep 17 23:15:21.607250 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.607141 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f4d315-2130-4dc1-8c78-fca47740b436-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-x5r9q\" (UID: \"c2f4d315-2130-4dc1-8c78-fca47740b436\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" Sep 17 23:15:21.608012 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.607989 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f4d315-2130-4dc1-8c78-fca47740b436-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-x5r9q\" (UID: \"c2f4d315-2130-4dc1-8c78-fca47740b436\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" Sep 17 23:15:21.621210 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.621183 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2gt\" (UniqueName: \"kubernetes.io/projected/c2f4d315-2130-4dc1-8c78-fca47740b436-kube-api-access-7t2gt\") pod \"payload-processing-5b54b7fbb7-x5r9q\" (UID: \"c2f4d315-2130-4dc1-8c78-fca47740b436\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" Sep 17 23:15:21.721870 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.721842 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" Sep 17 23:15:21.994601 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:21.994577 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-x5r9q"] Sep 17 23:15:21.997334 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:15:21.997306 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f4d315_2130_4dc1_8c78_fca47740b436.slice/crio-bb6ac0e8189260cd74e77edae4aa7efc3f2202fcea1a91e04197eac031145333 WatchSource:0}: Error finding container bb6ac0e8189260cd74e77edae4aa7efc3f2202fcea1a91e04197eac031145333: Status 404 returned error can't find the container with id bb6ac0e8189260cd74e77edae4aa7efc3f2202fcea1a91e04197eac031145333 Sep 17 23:15:22.012164 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:22.012133 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" event={"ID":"c2f4d315-2130-4dc1-8c78-fca47740b436","Type":"ContainerStarted","Data":"bb6ac0e8189260cd74e77edae4aa7efc3f2202fcea1a91e04197eac031145333"} Sep 17 23:15:25.028137 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:25.028090 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" event={"ID":"c2f4d315-2130-4dc1-8c78-fca47740b436","Type":"ContainerStarted","Data":"c6dd01aaf2a1e1bd927d688f0d5913cbef3d652160ba6efdc600659761bbe260"} Sep 17 23:15:25.028525 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:25.028165 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" Sep 17 23:15:25.045479 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:25.045425 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" podStartSLOduration=1.683041507 podStartE2EDuration="4.045410844s" totalImagesPullingTime="2.362369337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:25.04436234 +0000 UTC m=+553.161841808" watchObservedRunningTime="2026-09-17 23:15:25.045410844 +0000 UTC m=+553.162890285" Sep 17 23:15:36.035946 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:36.035894 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-x5r9q" Sep 17 23:15:53.018889 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:15:53.018851 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:16:04.091141 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:04.091099 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:16:09.890435 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.890400 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6"] Sep 17 23:16:09.892890 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.892869 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:09.895245 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.895227 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 17 23:16:09.896158 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.896141 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 17 23:16:09.896255 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.896207 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wkppn\"" Sep 17 23:16:09.896255 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.896245 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 17 23:16:09.903742 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.903719 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6"] Sep 17 23:16:09.975890 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.975862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833c7bb0-3e22-405d-af9f-bb8cf93ef707-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:09.975890 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.975896 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833c7bb0-3e22-405d-af9f-bb8cf93ef707-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:09.976083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.975956 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833c7bb0-3e22-405d-af9f-bb8cf93ef707-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:09.976083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.975974 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833c7bb0-3e22-405d-af9f-bb8cf93ef707-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:09.976083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.976010 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2mpd\" (UniqueName: \"kubernetes.io/projected/833c7bb0-3e22-405d-af9f-bb8cf93ef707-kube-api-access-j2mpd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:09.976083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:09.976063 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833c7bb0-3e22-405d-af9f-bb8cf93ef707-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.077189 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.077156 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2mpd\" (UniqueName: \"kubernetes.io/projected/833c7bb0-3e22-405d-af9f-bb8cf93ef707-kube-api-access-j2mpd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.077336 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.077203 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833c7bb0-3e22-405d-af9f-bb8cf93ef707-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.077336 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.077234 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833c7bb0-3e22-405d-af9f-bb8cf93ef707-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.077336 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.077254 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833c7bb0-3e22-405d-af9f-bb8cf93ef707-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.077336 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.077286 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833c7bb0-3e22-405d-af9f-bb8cf93ef707-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.077336 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.077301 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833c7bb0-3e22-405d-af9f-bb8cf93ef707-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.077771 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.077749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833c7bb0-3e22-405d-af9f-bb8cf93ef707-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.077848 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.077785 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833c7bb0-3e22-405d-af9f-bb8cf93ef707-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.077848 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.077817 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833c7bb0-3e22-405d-af9f-bb8cf93ef707-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.079454 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.079430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833c7bb0-3e22-405d-af9f-bb8cf93ef707-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.079694 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.079676 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833c7bb0-3e22-405d-af9f-bb8cf93ef707-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.085295 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.085277 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2mpd\" (UniqueName: \"kubernetes.io/projected/833c7bb0-3e22-405d-af9f-bb8cf93ef707-kube-api-access-j2mpd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6\" (UID: \"833c7bb0-3e22-405d-af9f-bb8cf93ef707\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.203946 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.203894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:10.475163 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.475141 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6"] Sep 17 23:16:10.477382 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:16:10.477354 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833c7bb0_3e22_405d_af9f_bb8cf93ef707.slice/crio-7aaa4ddabf33d155ebb8cae9073734298985790b7d98588e84f0e7d85a7753c0 WatchSource:0}: Error finding container 7aaa4ddabf33d155ebb8cae9073734298985790b7d98588e84f0e7d85a7753c0: Status 404 returned error can't find the container with id 7aaa4ddabf33d155ebb8cae9073734298985790b7d98588e84f0e7d85a7753c0 Sep 17 23:16:10.804157 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:10.804078 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:16:11.248135 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:11.248064 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" event={"ID":"833c7bb0-3e22-405d-af9f-bb8cf93ef707","Type":"ContainerStarted","Data":"7aaa4ddabf33d155ebb8cae9073734298985790b7d98588e84f0e7d85a7753c0"} Sep 17 23:16:12.254513 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:12.254478 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" event={"ID":"833c7bb0-3e22-405d-af9f-bb8cf93ef707","Type":"ContainerStarted","Data":"87d3470da3e08b5ad15ffbd088418c23ff438ac7bf7e49f72826bacb7ed4f6fb"} Sep 17 23:16:12.272083 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:12.272038 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" podStartSLOduration=2.360890355 podStartE2EDuration="3.272024104s" totalImagesPullingTime="911.133749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:12.270583007 +0000 UTC m=+600.388062456" watchObservedRunningTime="2026-09-17 23:16:12.272024104 +0000 UTC m=+600.389503553" Sep 17 23:16:14.187294 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:14.187263 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:16:14.187646 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:14.187394 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:16:20.204473 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:20.204439 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:20.206054 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:20.204506 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:20.216189 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:20.216172 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:20.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:20.303477 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6" Sep 17 23:16:28.597240 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:28.597163 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:16:34.403315 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:34.403283 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:16:49.406816 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:16:49.406782 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:17:00.610496 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:17:00.610463 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:18:35.958560 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:35.958524 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7862-c8a2b7"] Sep 17 23:18:35.961957 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:35.961937 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-c8a2b7" Sep 17 23:18:35.964403 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:35.964380 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pml6m\"" Sep 17 23:18:35.966119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:35.966097 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7862-c8a2b7"] Sep 17 23:18:36.061224 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:36.061188 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrpc\" (UniqueName: \"kubernetes.io/projected/48f49710-292d-4434-b45f-dac63db2dcd6-kube-api-access-7wrpc\") pod \"test-curl-7862-c8a2b7\" (UID: \"48f49710-292d-4434-b45f-dac63db2dcd6\") " pod="opendatahub/test-curl-7862-c8a2b7" Sep 17 23:18:36.162003 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:36.161968 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrpc\" (UniqueName: \"kubernetes.io/projected/48f49710-292d-4434-b45f-dac63db2dcd6-kube-api-access-7wrpc\") pod \"test-curl-7862-c8a2b7\" (UID: \"48f49710-292d-4434-b45f-dac63db2dcd6\") " pod="opendatahub/test-curl-7862-c8a2b7" Sep 17 23:18:36.169147 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:36.169117 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrpc\" (UniqueName: \"kubernetes.io/projected/48f49710-292d-4434-b45f-dac63db2dcd6-kube-api-access-7wrpc\") pod \"test-curl-7862-c8a2b7\" (UID: \"48f49710-292d-4434-b45f-dac63db2dcd6\") " pod="opendatahub/test-curl-7862-c8a2b7" Sep 17 23:18:36.273317 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:36.273250 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-c8a2b7" Sep 17 23:18:36.545208 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:36.545180 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7862-c8a2b7"] Sep 17 23:18:36.547061 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:18:36.547035 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48f49710_292d_4434_b45f_dac63db2dcd6.slice/crio-828d9525015fc7b0a7f92853f3e2c74f78d4e364bb2f864d5dbf5c86e32cfd94 WatchSource:0}: Error finding container 828d9525015fc7b0a7f92853f3e2c74f78d4e364bb2f864d5dbf5c86e32cfd94: Status 404 returned error can't find the container with id 828d9525015fc7b0a7f92853f3e2c74f78d4e364bb2f864d5dbf5c86e32cfd94 Sep 17 23:18:36.548741 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:36.548722 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 23:18:36.939031 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:36.938946 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7862-c8a2b7" event={"ID":"48f49710-292d-4434-b45f-dac63db2dcd6","Type":"ContainerStarted","Data":"828d9525015fc7b0a7f92853f3e2c74f78d4e364bb2f864d5dbf5c86e32cfd94"} Sep 17 23:18:37.717525 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:37.717489 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:18:39.646967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.646937 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk"] Sep 17 23:18:39.650376 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.650357 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" Sep 17 23:18:39.653197 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.653169 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-825f30-dockercfg-9fxxk\"" Sep 17 23:18:39.653312 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.653271 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-825f30\"" Sep 17 23:18:39.665082 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.665056 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk"] Sep 17 23:18:39.668158 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.668132 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc"] Sep 17 23:18:39.672142 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.672114 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" Sep 17 23:18:39.685005 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.684984 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc"] Sep 17 23:18:39.796042 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.796012 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2e17bb-783b-49ee-9104-f4fde4ce610c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk\" (UID: \"cc2e17bb-783b-49ee-9104-f4fde4ce610c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" Sep 17 23:18:39.796176 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.796059 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc\" (UID: \"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" Sep 17 23:18:39.796176 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.796162 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pp24\" (UniqueName: \"kubernetes.io/projected/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9-kube-api-access-7pp24\") pod \"payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc\" (UID: \"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" Sep 17 23:18:39.796256 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.796198 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2lg\" (UniqueName: \"kubernetes.io/projected/cc2e17bb-783b-49ee-9104-f4fde4ce610c-kube-api-access-xd2lg\") pod \"payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk\" (UID: \"cc2e17bb-783b-49ee-9104-f4fde4ce610c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" Sep 17 23:18:39.897635 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.897556 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2e17bb-783b-49ee-9104-f4fde4ce610c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk\" (UID: \"cc2e17bb-783b-49ee-9104-f4fde4ce610c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" Sep 17 23:18:39.897788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.897708 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc\" (UID: \"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" Sep 17 23:18:39.897851 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.897837 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pp24\" (UniqueName: \"kubernetes.io/projected/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9-kube-api-access-7pp24\") pod \"payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc\" (UID: \"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" Sep 17 23:18:39.897919 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.897874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2lg\" (UniqueName: \"kubernetes.io/projected/cc2e17bb-783b-49ee-9104-f4fde4ce610c-kube-api-access-xd2lg\") pod \"payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk\" (UID: \"cc2e17bb-783b-49ee-9104-f4fde4ce610c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" Sep 17 23:18:39.898454 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.898430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc\" (UID: \"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" Sep 17 23:18:39.898554 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.898517 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2e17bb-783b-49ee-9104-f4fde4ce610c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk\" (UID: \"cc2e17bb-783b-49ee-9104-f4fde4ce610c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" Sep 17 23:18:39.910725 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.910699 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2lg\" (UniqueName: \"kubernetes.io/projected/cc2e17bb-783b-49ee-9104-f4fde4ce610c-kube-api-access-xd2lg\") pod \"payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk\" (UID: \"cc2e17bb-783b-49ee-9104-f4fde4ce610c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" Sep 17 23:18:39.911365 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.911345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pp24\" (UniqueName: \"kubernetes.io/projected/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9-kube-api-access-7pp24\") pod \"payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc\" (UID: \"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" Sep 17 23:18:39.961453 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.961420 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" Sep 17 23:18:39.962334 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.962305 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7862-c8a2b7" event={"ID":"48f49710-292d-4434-b45f-dac63db2dcd6","Type":"ContainerStarted","Data":"07bcc00ff56dee7259bd25b30b640c65d8e6aba3d0a630511cbfb8e14f93dbf8"} Sep 17 23:18:39.982287 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.982261 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" Sep 17 23:18:39.982436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:39.982388 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7862-c8a2b7" podStartSLOduration=2.423049339 podStartE2EDuration="4.982368145s" totalImagesPullingTime="2.559318806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:39.980270382 +0000 UTC m=+748.097749824" watchObservedRunningTime="2026-09-17 23:18:39.982368145 +0000 UTC m=+748.099847597" Sep 17 23:18:40.241691 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.241657 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk"] Sep 17 23:18:40.269956 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.269927 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc"] Sep 17 23:18:40.271510 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:18:40.271482 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ae6ed0_cfc8_4bfc_8880_5ef0db85a9a9.slice/crio-e83ade0937dcb50d9628bed522f18bef7ebf35c79407691625bcb3a926190171 WatchSource:0}: Error finding container e83ade0937dcb50d9628bed522f18bef7ebf35c79407691625bcb3a926190171: Status 404 returned error can't find the container with id e83ade0937dcb50d9628bed522f18bef7ebf35c79407691625bcb3a926190171 Sep 17 23:18:40.538593 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.538516 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7862-c8a2b7"] Sep 17 23:18:40.544938 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.542203 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7862-c8a2b7"] Sep 17 23:18:40.559613 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.559580 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx"] Sep 17 23:18:40.560238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.560216 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f49710-292d-4434-b45f-dac63db2dcd6" containerName="test-curl-7862-c8a2b7" Sep 17 23:18:40.560331 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.560241 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f49710-292d-4434-b45f-dac63db2dcd6" containerName="test-curl-7862-c8a2b7" Sep 17 23:18:40.560393 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.560348 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="48f49710-292d-4434-b45f-dac63db2dcd6" containerName="test-curl-7862-c8a2b7" Sep 17 23:18:40.563802 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.563785 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" Sep 17 23:18:40.566435 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.566306 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-48d98f-dockercfg-zps7v\"" Sep 17 23:18:40.566544 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.566532 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-48d98f\"" Sep 17 23:18:40.571815 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.571793 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j"] Sep 17 23:18:40.575479 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.575452 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" Sep 17 23:18:40.580210 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.580191 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx"] Sep 17 23:18:40.585991 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.585966 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j"] Sep 17 23:18:40.657845 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.657812 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7862-9ee82a"] Sep 17 23:18:40.662024 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.661999 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:40.669236 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.669166 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7862-9ee82a"] Sep 17 23:18:40.705630 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.705601 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rtfl\" (UniqueName: \"kubernetes.io/projected/b9e66f44-da70-4361-a106-a2a731870f47-kube-api-access-8rtfl\") pod \"payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j\" (UID: \"b9e66f44-da70-4361-a106-a2a731870f47\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" Sep 17 23:18:40.705798 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.705679 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be52dd4-470a-437c-9ad9-a749a110efe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx\" (UID: \"7be52dd4-470a-437c-9ad9-a749a110efe6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" Sep 17 23:18:40.705798 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.705717 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e66f44-da70-4361-a106-a2a731870f47-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j\" (UID: \"b9e66f44-da70-4361-a106-a2a731870f47\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" Sep 17 23:18:40.705798 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.705763 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fk2v\" (UniqueName: \"kubernetes.io/projected/7be52dd4-470a-437c-9ad9-a749a110efe6-kube-api-access-8fk2v\") pod \"payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx\" (UID: \"7be52dd4-470a-437c-9ad9-a749a110efe6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" Sep 17 23:18:40.806593 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.806493 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be52dd4-470a-437c-9ad9-a749a110efe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx\" (UID: \"7be52dd4-470a-437c-9ad9-a749a110efe6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" Sep 17 23:18:40.806593 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.806538 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e66f44-da70-4361-a106-a2a731870f47-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j\" (UID: \"b9e66f44-da70-4361-a106-a2a731870f47\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" Sep 17 23:18:40.806593 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.806566 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fk2v\" (UniqueName: \"kubernetes.io/projected/7be52dd4-470a-437c-9ad9-a749a110efe6-kube-api-access-8fk2v\") pod \"payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx\" (UID: \"7be52dd4-470a-437c-9ad9-a749a110efe6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" Sep 17 23:18:40.806859 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.806680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfkw7\" (UniqueName: \"kubernetes.io/projected/53c76451-4ff3-4530-b4e6-9f7c8c112bc8-kube-api-access-vfkw7\") pod \"test-curl-7862-9ee82a\" (UID: \"53c76451-4ff3-4530-b4e6-9f7c8c112bc8\") " pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:40.806859 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.806716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rtfl\" (UniqueName: \"kubernetes.io/projected/b9e66f44-da70-4361-a106-a2a731870f47-kube-api-access-8rtfl\") pod \"payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j\" (UID: \"b9e66f44-da70-4361-a106-a2a731870f47\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" Sep 17 23:18:40.807330 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.807308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e66f44-da70-4361-a106-a2a731870f47-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j\" (UID: \"b9e66f44-da70-4361-a106-a2a731870f47\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" Sep 17 23:18:40.807384 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.807314 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be52dd4-470a-437c-9ad9-a749a110efe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx\" (UID: \"7be52dd4-470a-437c-9ad9-a749a110efe6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" Sep 17 23:18:40.814707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.814680 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rtfl\" (UniqueName: \"kubernetes.io/projected/b9e66f44-da70-4361-a106-a2a731870f47-kube-api-access-8rtfl\") pod \"payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j\" (UID: \"b9e66f44-da70-4361-a106-a2a731870f47\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" Sep 17 23:18:40.814822 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.814759 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fk2v\" (UniqueName: \"kubernetes.io/projected/7be52dd4-470a-437c-9ad9-a749a110efe6-kube-api-access-8fk2v\") pod \"payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx\" (UID: \"7be52dd4-470a-437c-9ad9-a749a110efe6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" Sep 17 23:18:40.880406 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.880370 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" Sep 17 23:18:40.890595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.890573 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" Sep 17 23:18:40.907894 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.907865 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfkw7\" (UniqueName: \"kubernetes.io/projected/53c76451-4ff3-4530-b4e6-9f7c8c112bc8-kube-api-access-vfkw7\") pod \"test-curl-7862-9ee82a\" (UID: \"53c76451-4ff3-4530-b4e6-9f7c8c112bc8\") " pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:40.918772 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.918719 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfkw7\" (UniqueName: \"kubernetes.io/projected/53c76451-4ff3-4530-b4e6-9f7c8c112bc8-kube-api-access-vfkw7\") pod \"test-curl-7862-9ee82a\" (UID: \"53c76451-4ff3-4530-b4e6-9f7c8c112bc8\") " pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:40.971415 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.971335 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" event={"ID":"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9","Type":"ContainerStarted","Data":"30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b"} Sep 17 23:18:40.971415 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.971385 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" event={"ID":"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9","Type":"ContainerStarted","Data":"e83ade0937dcb50d9628bed522f18bef7ebf35c79407691625bcb3a926190171"} Sep 17 23:18:40.971639 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.971442 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" Sep 17 23:18:40.973069 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.973043 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" event={"ID":"cc2e17bb-783b-49ee-9104-f4fde4ce610c","Type":"ContainerStarted","Data":"7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d"} Sep 17 23:18:40.973167 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.973087 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" event={"ID":"cc2e17bb-783b-49ee-9104-f4fde4ce610c","Type":"ContainerStarted","Data":"69a1125108f4579725574b06d5071b0ad752e91691d97d62cfc9db9425e36639"} Sep 17 23:18:40.973222 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:18:40.973193 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7862-c8a2b7" container="test-curl-7862-c8a2b7" Sep 17 23:18:40.974798 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.974780 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:40.990015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:40.989894 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" podStartSLOduration=1.989875697 podStartE2EDuration="1.989875697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:40.987784385 +0000 UTC m=+749.105263837" watchObservedRunningTime="2026-09-17 23:18:40.989875697 +0000 UTC m=+749.107355151" Sep 17 23:18:41.009368 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.009177 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" podStartSLOduration=2.009161555 podStartE2EDuration="2.009161555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:41.007387198 +0000 UTC m=+749.124866647" watchObservedRunningTime="2026-09-17 23:18:41.009161555 +0000 UTC m=+749.126641005" Sep 17 23:18:41.158671 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.158628 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx"] Sep 17 23:18:41.160676 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:18:41.160645 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be52dd4_470a_437c_9ad9_a749a110efe6.slice/crio-58a12cabf67d8d06bab46e96c7dcbf3b1e29a71b492a10671701f19760751e99 WatchSource:0}: Error finding container 58a12cabf67d8d06bab46e96c7dcbf3b1e29a71b492a10671701f19760751e99: Status 404 returned error can't find the container with id 58a12cabf67d8d06bab46e96c7dcbf3b1e29a71b492a10671701f19760751e99 Sep 17 23:18:41.176158 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.176123 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j"] Sep 17 23:18:41.177343 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:18:41.177318 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e66f44_da70_4361_a106_a2a731870f47.slice/crio-e1c39c020737a45bdb0d82d33771d97736c845b4de64e5c6ed3cad76987255f8 WatchSource:0}: Error finding container e1c39c020737a45bdb0d82d33771d97736c845b4de64e5c6ed3cad76987255f8: Status 404 returned error can't find the container with id e1c39c020737a45bdb0d82d33771d97736c845b4de64e5c6ed3cad76987255f8 Sep 17 23:18:41.264763 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.264698 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7862-9ee82a"] Sep 17 23:18:41.266892 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:18:41.266848 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53c76451_4ff3_4530_b4e6_9f7c8c112bc8.slice/crio-9beaf2b9476794253cd50a4e0a56f7c85867466b39bb5a1a15f5d5f25d8baddc WatchSource:0}: Error finding container 9beaf2b9476794253cd50a4e0a56f7c85867466b39bb5a1a15f5d5f25d8baddc: Status 404 returned error can't find the container with id 9beaf2b9476794253cd50a4e0a56f7c85867466b39bb5a1a15f5d5f25d8baddc Sep 17 23:18:41.979947 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.979895 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7862-9ee82a" event={"ID":"53c76451-4ff3-4530-b4e6-9f7c8c112bc8","Type":"ContainerStarted","Data":"a1daa123626dbaf2dfdc01aab4d9d5e7174ad8d1b9df595ddf1ea499031bf3d5"} Sep 17 23:18:41.980393 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.979958 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7862-9ee82a" event={"ID":"53c76451-4ff3-4530-b4e6-9f7c8c112bc8","Type":"ContainerStarted","Data":"9beaf2b9476794253cd50a4e0a56f7c85867466b39bb5a1a15f5d5f25d8baddc"} Sep 17 23:18:41.981630 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.981603 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" event={"ID":"7be52dd4-470a-437c-9ad9-a749a110efe6","Type":"ContainerStarted","Data":"72cfc9c38283e479650595986f226ca84f9dbf793b8da467d72c98ebee180055"} Sep 17 23:18:41.981786 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.981650 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" event={"ID":"7be52dd4-470a-437c-9ad9-a749a110efe6","Type":"ContainerStarted","Data":"58a12cabf67d8d06bab46e96c7dcbf3b1e29a71b492a10671701f19760751e99"} Sep 17 23:18:41.981786 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.981680 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" Sep 17 23:18:41.983264 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.983239 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" event={"ID":"b9e66f44-da70-4361-a106-a2a731870f47","Type":"ContainerStarted","Data":"2422201a987441049f8a459a888b41193f88ca7653f58a0a2a16071f67386366"} Sep 17 23:18:41.983436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.983353 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7862-c8a2b7" podUID="48f49710-292d-4434-b45f-dac63db2dcd6" containerName="test-curl-7862-c8a2b7" containerID="cri-o://07bcc00ff56dee7259bd25b30b640c65d8e6aba3d0a630511cbfb8e14f93dbf8" gracePeriod=2 Sep 17 23:18:41.983719 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.983696 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" Sep 17 23:18:41.983827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.983742 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" Sep 17 23:18:41.983827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.983766 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" event={"ID":"b9e66f44-da70-4361-a106-a2a731870f47","Type":"ContainerStarted","Data":"e1c39c020737a45bdb0d82d33771d97736c845b4de64e5c6ed3cad76987255f8"} Sep 17 23:18:41.995402 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:18:41.995376 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-c8a2b7\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="48f49710-292d-4434-b45f-dac63db2dcd6" pod="opendatahub/test-curl-7862-c8a2b7" Sep 17 23:18:41.995525 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:41.995457 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7862-9ee82a" podStartSLOduration=1.9954446049999999 podStartE2EDuration="1.995444605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:41.99340712 +0000 UTC m=+750.110886571" watchObservedRunningTime="2026-09-17 23:18:41.995444605 +0000 UTC m=+750.112924033" Sep 17 23:18:42.014116 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:42.014072 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" podStartSLOduration=2.014058716 podStartE2EDuration="2.014058716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:42.012126608 +0000 UTC m=+750.129606080" watchObservedRunningTime="2026-09-17 23:18:42.014058716 +0000 UTC m=+750.131538165" Sep 17 23:18:42.027316 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:42.027272 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" podStartSLOduration=2.027256089 podStartE2EDuration="2.027256089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:42.026930235 +0000 UTC m=+750.144409704" watchObservedRunningTime="2026-09-17 23:18:42.027256089 +0000 UTC m=+750.144735539" Sep 17 23:18:42.597932 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:42.597888 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7862-9ee82a"] Sep 17 23:18:42.604116 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:42.604090 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7862-9ee82a"] Sep 17 23:18:42.988914 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:18:42.988882 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7862-9ee82a" container="test-curl-7862-9ee82a" Sep 17 23:18:43.611502 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.611467 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r"] Sep 17 23:18:43.612092 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.612076 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c76451-4ff3-4530-b4e6-9f7c8c112bc8" containerName="test-curl-7862-9ee82a" Sep 17 23:18:43.612163 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.612095 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c76451-4ff3-4530-b4e6-9f7c8c112bc8" containerName="test-curl-7862-9ee82a" Sep 17 23:18:43.612216 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.612206 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c76451-4ff3-4530-b4e6-9f7c8c112bc8" containerName="test-curl-7862-9ee82a" Sep 17 23:18:43.615970 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.615951 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.618485 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.618463 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 23:18:43.618607 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.618463 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-54be2504-openshift-default-dockercfg-vmnf7\"" Sep 17 23:18:43.624783 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.624757 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r"] Sep 17 23:18:43.734195 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.734165 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-credential-socket\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.734195 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.734199 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b64dl\" (UniqueName: \"kubernetes.io/projected/3ff0c6ea-5409-428a-a22c-4447438235a8-kube-api-access-b64dl\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.734403 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.734221 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-podinfo\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.734403 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.734276 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-envoy\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.734403 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.734383 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff0c6ea-5409-428a-a22c-4447438235a8-istiod-ca-cert\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.734505 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.734428 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-workload-socket\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.734505 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.734455 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-workload-certs\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.734505 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.734483 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-data\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.734623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.734517 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-token\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.835470 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.835431 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-token\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.835617 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.835519 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-credential-socket\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.835617 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.835543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b64dl\" (UniqueName: \"kubernetes.io/projected/3ff0c6ea-5409-428a-a22c-4447438235a8-kube-api-access-b64dl\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.835617 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.835570 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-podinfo\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.835617 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.835598 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-envoy\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.835838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.835660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff0c6ea-5409-428a-a22c-4447438235a8-istiod-ca-cert\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.835838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.835698 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-workload-socket\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.835838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.835723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-workload-certs\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.835838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.835752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-data\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.836069 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.835975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-credential-socket\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.836249 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.836226 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-workload-socket\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.836358 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.836333 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-workload-certs\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.836358 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.836351 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-data\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.836609 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.836577 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff0c6ea-5409-428a-a22c-4447438235a8-istiod-ca-cert\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.838455 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.838433 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-envoy\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.838658 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.838642 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-podinfo\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.844335 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.844307 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-token\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.844335 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.844334 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b64dl\" (UniqueName: \"kubernetes.io/projected/3ff0c6ea-5409-428a-a22c-4447438235a8-kube-api-access-b64dl\") pod \"e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.930111 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.930052 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:43.999964 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:43.999651 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7862-9ee82a" podUID="53c76451-4ff3-4530-b4e6-9f7c8c112bc8" containerName="test-curl-7862-9ee82a" containerID="cri-o://a1daa123626dbaf2dfdc01aab4d9d5e7174ad8d1b9df595ddf1ea499031bf3d5" gracePeriod=2 Sep 17 23:18:44.003681 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:18:44.003647 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-9ee82a\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="53c76451-4ff3-4530-b4e6-9f7c8c112bc8" pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:44.123804 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:44.123779 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-c8a2b7" Sep 17 23:18:44.125837 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:18:44.125811 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-9ee82a\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="53c76451-4ff3-4530-b4e6-9f7c8c112bc8" pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:44.210486 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:44.210462 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r"] Sep 17 23:18:44.212412 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:18:44.212390 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff0c6ea_5409_428a_a22c_4447438235a8.slice/crio-99a621a2405be0a37b2d50ff702acb249e7beb379395a0dd9068cf98f9f72caa WatchSource:0}: Error finding container 99a621a2405be0a37b2d50ff702acb249e7beb379395a0dd9068cf98f9f72caa: Status 404 returned error can't find the container with id 99a621a2405be0a37b2d50ff702acb249e7beb379395a0dd9068cf98f9f72caa Sep 17 23:18:44.239188 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:44.239163 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wrpc\" (UniqueName: \"kubernetes.io/projected/48f49710-292d-4434-b45f-dac63db2dcd6-kube-api-access-7wrpc\") pod \"48f49710-292d-4434-b45f-dac63db2dcd6\" (UID: \"48f49710-292d-4434-b45f-dac63db2dcd6\") " Sep 17 23:18:44.241641 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:44.241605 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f49710-292d-4434-b45f-dac63db2dcd6-kube-api-access-7wrpc" (OuterVolumeSpecName: "kube-api-access-7wrpc") pod "48f49710-292d-4434-b45f-dac63db2dcd6" (UID: "48f49710-292d-4434-b45f-dac63db2dcd6"). InnerVolumeSpecName "kube-api-access-7wrpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:18:44.284635 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:18:44.284601 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-9ee82a\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="53c76451-4ff3-4530-b4e6-9f7c8c112bc8" pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:44.286432 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:44.286411 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48f49710-292d-4434-b45f-dac63db2dcd6" path="/var/lib/kubelet/pods/48f49710-292d-4434-b45f-dac63db2dcd6/volumes" Sep 17 23:18:44.340868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:44.340844 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wrpc\" (UniqueName: \"kubernetes.io/projected/48f49710-292d-4434-b45f-dac63db2dcd6-kube-api-access-7wrpc\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:18:45.008569 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:45.008523 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" event={"ID":"3ff0c6ea-5409-428a-a22c-4447438235a8","Type":"ContainerStarted","Data":"99a621a2405be0a37b2d50ff702acb249e7beb379395a0dd9068cf98f9f72caa"} Sep 17 23:18:45.010748 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:45.010713 2559 generic.go:356] "Generic (PLEG): container finished" podID="48f49710-292d-4434-b45f-dac63db2dcd6" containerID="07bcc00ff56dee7259bd25b30b640c65d8e6aba3d0a630511cbfb8e14f93dbf8" exitCode=137 Sep 17 23:18:45.010889 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:45.010793 2559 scope.go:117] "RemoveContainer" containerID="07bcc00ff56dee7259bd25b30b640c65d8e6aba3d0a630511cbfb8e14f93dbf8" Sep 17 23:18:45.010967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:45.010960 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-c8a2b7" Sep 17 23:18:45.021306 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:45.021253 2559 scope.go:117] "RemoveContainer" containerID="07bcc00ff56dee7259bd25b30b640c65d8e6aba3d0a630511cbfb8e14f93dbf8" Sep 17 23:18:45.021629 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:18:45.021605 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bcc00ff56dee7259bd25b30b640c65d8e6aba3d0a630511cbfb8e14f93dbf8\": container with ID starting with 07bcc00ff56dee7259bd25b30b640c65d8e6aba3d0a630511cbfb8e14f93dbf8 not found: ID does not exist" containerID="07bcc00ff56dee7259bd25b30b640c65d8e6aba3d0a630511cbfb8e14f93dbf8" Sep 17 23:18:45.021737 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:45.021640 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bcc00ff56dee7259bd25b30b640c65d8e6aba3d0a630511cbfb8e14f93dbf8"} err="failed to get container status \"07bcc00ff56dee7259bd25b30b640c65d8e6aba3d0a630511cbfb8e14f93dbf8\": rpc error: code = NotFound desc = could not find container \"07bcc00ff56dee7259bd25b30b640c65d8e6aba3d0a630511cbfb8e14f93dbf8\": container with ID starting with 07bcc00ff56dee7259bd25b30b640c65d8e6aba3d0a630511cbfb8e14f93dbf8 not found: ID does not exist" Sep 17 23:18:46.745590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:46.745464 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:18:46.745590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:46.745563 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:18:46.746080 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:46.745606 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:18:46.821039 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:46.821015 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:46.967241 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:46.967209 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfkw7\" (UniqueName: \"kubernetes.io/projected/53c76451-4ff3-4530-b4e6-9f7c8c112bc8-kube-api-access-vfkw7\") pod \"53c76451-4ff3-4530-b4e6-9f7c8c112bc8\" (UID: \"53c76451-4ff3-4530-b4e6-9f7c8c112bc8\") " Sep 17 23:18:46.969323 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:46.969293 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c76451-4ff3-4530-b4e6-9f7c8c112bc8-kube-api-access-vfkw7" (OuterVolumeSpecName: "kube-api-access-vfkw7") pod "53c76451-4ff3-4530-b4e6-9f7c8c112bc8" (UID: "53c76451-4ff3-4530-b4e6-9f7c8c112bc8"). InnerVolumeSpecName "kube-api-access-vfkw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:18:47.024478 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:47.024446 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" event={"ID":"3ff0c6ea-5409-428a-a22c-4447438235a8","Type":"ContainerStarted","Data":"a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5"} Sep 17 23:18:47.025987 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:47.025933 2559 generic.go:356] "Generic (PLEG): container finished" podID="53c76451-4ff3-4530-b4e6-9f7c8c112bc8" containerID="a1daa123626dbaf2dfdc01aab4d9d5e7174ad8d1b9df595ddf1ea499031bf3d5" exitCode=137 Sep 17 23:18:47.026169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:47.025986 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:47.026169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:47.026017 2559 scope.go:117] "RemoveContainer" containerID="a1daa123626dbaf2dfdc01aab4d9d5e7174ad8d1b9df595ddf1ea499031bf3d5" Sep 17 23:18:47.035503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:47.035486 2559 scope.go:117] "RemoveContainer" containerID="a1daa123626dbaf2dfdc01aab4d9d5e7174ad8d1b9df595ddf1ea499031bf3d5" Sep 17 23:18:47.035768 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:18:47.035751 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1daa123626dbaf2dfdc01aab4d9d5e7174ad8d1b9df595ddf1ea499031bf3d5\": container with ID starting with a1daa123626dbaf2dfdc01aab4d9d5e7174ad8d1b9df595ddf1ea499031bf3d5 not found: ID does not exist" containerID="a1daa123626dbaf2dfdc01aab4d9d5e7174ad8d1b9df595ddf1ea499031bf3d5" Sep 17 23:18:47.035844 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:47.035776 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1daa123626dbaf2dfdc01aab4d9d5e7174ad8d1b9df595ddf1ea499031bf3d5"} err="failed to get container status \"a1daa123626dbaf2dfdc01aab4d9d5e7174ad8d1b9df595ddf1ea499031bf3d5\": rpc error: code = NotFound desc = could not find container \"a1daa123626dbaf2dfdc01aab4d9d5e7174ad8d1b9df595ddf1ea499031bf3d5\": container with ID starting with a1daa123626dbaf2dfdc01aab4d9d5e7174ad8d1b9df595ddf1ea499031bf3d5 not found: ID does not exist" Sep 17 23:18:47.049633 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:47.049582 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" podStartSLOduration=1.5188780990000001 podStartE2EDuration="4.049566867s" totalImagesPullingTime="2.530688768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:47.048278018 +0000 UTC m=+755.165757513" watchObservedRunningTime="2026-09-17 23:18:47.049566867 +0000 UTC m=+755.167046317" Sep 17 23:18:47.050316 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:18:47.050279 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-9ee82a\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="53c76451-4ff3-4530-b4e6-9f7c8c112bc8" pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:47.068833 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:47.068804 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfkw7\" (UniqueName: \"kubernetes.io/projected/53c76451-4ff3-4530-b4e6-9f7c8c112bc8-kube-api-access-vfkw7\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:18:47.931179 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:47.931148 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:47.936084 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:47.936060 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:47.938095 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:18:47.938069 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-9ee82a\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="53c76451-4ff3-4530-b4e6-9f7c8c112bc8" pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:48.032817 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:48.032786 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:48.033584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:48.033563 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:18:48.035461 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:18:48.035437 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-9ee82a\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="53c76451-4ff3-4530-b4e6-9f7c8c112bc8" pod="opendatahub/test-curl-7862-9ee82a" Sep 17 23:18:48.286631 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:48.286596 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c76451-4ff3-4530-b4e6-9f7c8c112bc8" path="/var/lib/kubelet/pods/53c76451-4ff3-4530-b4e6-9f7c8c112bc8/volumes" Sep 17 23:18:51.744971 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:51.744935 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4"] Sep 17 23:18:51.749701 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:51.749678 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" Sep 17 23:18:51.752104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:51.752083 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-61e623-dockercfg-kcn6c\"" Sep 17 23:18:51.752227 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:51.752103 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-61e623\"" Sep 17 23:18:51.759946 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:51.759921 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4"] Sep 17 23:18:51.805865 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:51.805834 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0745ffc0-3e9a-443d-94f0-ae7a3f72e550-plugins-config-volume\") pod \"payload-processing-e2e-autores-61e623-54fc96f859-lxvv4\" (UID: \"0745ffc0-3e9a-443d-94f0-ae7a3f72e550\") " pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" Sep 17 23:18:51.806032 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:51.805872 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpbm5\" (UniqueName: \"kubernetes.io/projected/0745ffc0-3e9a-443d-94f0-ae7a3f72e550-kube-api-access-vpbm5\") pod \"payload-processing-e2e-autores-61e623-54fc96f859-lxvv4\" (UID: \"0745ffc0-3e9a-443d-94f0-ae7a3f72e550\") " pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" Sep 17 23:18:51.906698 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:51.906660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0745ffc0-3e9a-443d-94f0-ae7a3f72e550-plugins-config-volume\") pod \"payload-processing-e2e-autores-61e623-54fc96f859-lxvv4\" (UID: \"0745ffc0-3e9a-443d-94f0-ae7a3f72e550\") " pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" Sep 17 23:18:51.906852 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:51.906814 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbm5\" (UniqueName: \"kubernetes.io/projected/0745ffc0-3e9a-443d-94f0-ae7a3f72e550-kube-api-access-vpbm5\") pod \"payload-processing-e2e-autores-61e623-54fc96f859-lxvv4\" (UID: \"0745ffc0-3e9a-443d-94f0-ae7a3f72e550\") " pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" Sep 17 23:18:51.907457 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:51.907436 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0745ffc0-3e9a-443d-94f0-ae7a3f72e550-plugins-config-volume\") pod \"payload-processing-e2e-autores-61e623-54fc96f859-lxvv4\" (UID: \"0745ffc0-3e9a-443d-94f0-ae7a3f72e550\") " pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" Sep 17 23:18:51.914698 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:51.914674 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpbm5\" (UniqueName: \"kubernetes.io/projected/0745ffc0-3e9a-443d-94f0-ae7a3f72e550-kube-api-access-vpbm5\") pod \"payload-processing-e2e-autores-61e623-54fc96f859-lxvv4\" (UID: \"0745ffc0-3e9a-443d-94f0-ae7a3f72e550\") " pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" Sep 17 23:18:51.986290 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:51.986264 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" Sep 17 23:18:52.061302 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:52.061221 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" Sep 17 23:18:52.330864 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:52.330841 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4"] Sep 17 23:18:52.333152 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:18:52.333121 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0745ffc0_3e9a_443d_94f0_ae7a3f72e550.slice/crio-ecf16f48ed8da4ef0177ab56108327f4827cd25c24480c77fbb588c1f2057c20 WatchSource:0}: Error finding container ecf16f48ed8da4ef0177ab56108327f4827cd25c24480c77fbb588c1f2057c20: Status 404 returned error can't find the container with id ecf16f48ed8da4ef0177ab56108327f4827cd25c24480c77fbb588c1f2057c20 Sep 17 23:18:52.991602 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:52.991565 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" Sep 17 23:18:52.992067 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:52.991896 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" Sep 17 23:18:53.063504 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:53.063471 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" event={"ID":"0745ffc0-3e9a-443d-94f0-ae7a3f72e550","Type":"ContainerStarted","Data":"89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26"} Sep 17 23:18:53.063668 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:53.063551 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" Sep 17 23:18:53.063668 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:53.063576 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" event={"ID":"0745ffc0-3e9a-443d-94f0-ae7a3f72e550","Type":"ContainerStarted","Data":"ecf16f48ed8da4ef0177ab56108327f4827cd25c24480c77fbb588c1f2057c20"} Sep 17 23:18:53.141476 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:53.141408 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" podStartSLOduration=2.141386439 podStartE2EDuration="2.141386439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:53.140558533 +0000 UTC m=+761.258038009" watchObservedRunningTime="2026-09-17 23:18:53.141386439 +0000 UTC m=+761.258865894" Sep 17 23:18:54.004072 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:54.004040 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" Sep 17 23:18:55.573764 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.573722 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp"] Sep 17 23:18:55.577384 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.577363 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" Sep 17 23:18:55.579719 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.579695 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-54be2504\"" Sep 17 23:18:55.579719 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.579716 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-54be2504-dockercfg-svgwp\"" Sep 17 23:18:55.587537 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.587509 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp"] Sep 17 23:18:55.598408 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.598379 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49"] Sep 17 23:18:55.603512 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.603491 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" Sep 17 23:18:55.612738 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.612718 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49"] Sep 17 23:18:55.633353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.633328 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k876s\" (UniqueName: \"kubernetes.io/projected/39e40474-42f3-497b-9ecd-b75b4b9d1848-kube-api-access-k876s\") pod \"payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49\" (UID: \"39e40474-42f3-497b-9ecd-b75b4b9d1848\") " pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" Sep 17 23:18:55.633444 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.633380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22839e6-73a6-412c-af9f-7eb8cb8482b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp\" (UID: \"d22839e6-73a6-412c-af9f-7eb8cb8482b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" Sep 17 23:18:55.633479 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.633466 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7l8c\" (UniqueName: \"kubernetes.io/projected/d22839e6-73a6-412c-af9f-7eb8cb8482b5-kube-api-access-k7l8c\") pod \"payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp\" (UID: \"d22839e6-73a6-412c-af9f-7eb8cb8482b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" Sep 17 23:18:55.633529 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.633514 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e40474-42f3-497b-9ecd-b75b4b9d1848-plugins-config-volume\") pod \"payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49\" (UID: \"39e40474-42f3-497b-9ecd-b75b4b9d1848\") " pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" Sep 17 23:18:55.734049 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.733994 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7l8c\" (UniqueName: \"kubernetes.io/projected/d22839e6-73a6-412c-af9f-7eb8cb8482b5-kube-api-access-k7l8c\") pod \"payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp\" (UID: \"d22839e6-73a6-412c-af9f-7eb8cb8482b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" Sep 17 23:18:55.734049 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.734053 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e40474-42f3-497b-9ecd-b75b4b9d1848-plugins-config-volume\") pod \"payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49\" (UID: \"39e40474-42f3-497b-9ecd-b75b4b9d1848\") " pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" Sep 17 23:18:55.734233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.734077 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k876s\" (UniqueName: \"kubernetes.io/projected/39e40474-42f3-497b-9ecd-b75b4b9d1848-kube-api-access-k876s\") pod \"payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49\" (UID: \"39e40474-42f3-497b-9ecd-b75b4b9d1848\") " pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" Sep 17 23:18:55.734233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.734119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22839e6-73a6-412c-af9f-7eb8cb8482b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp\" (UID: \"d22839e6-73a6-412c-af9f-7eb8cb8482b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" Sep 17 23:18:55.734768 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.734747 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e40474-42f3-497b-9ecd-b75b4b9d1848-plugins-config-volume\") pod \"payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49\" (UID: \"39e40474-42f3-497b-9ecd-b75b4b9d1848\") " pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" Sep 17 23:18:55.734803 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.734755 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22839e6-73a6-412c-af9f-7eb8cb8482b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp\" (UID: \"d22839e6-73a6-412c-af9f-7eb8cb8482b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" Sep 17 23:18:55.745364 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.745327 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k876s\" (UniqueName: \"kubernetes.io/projected/39e40474-42f3-497b-9ecd-b75b4b9d1848-kube-api-access-k876s\") pod \"payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49\" (UID: \"39e40474-42f3-497b-9ecd-b75b4b9d1848\") " pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" Sep 17 23:18:55.745495 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.745395 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7l8c\" (UniqueName: \"kubernetes.io/projected/d22839e6-73a6-412c-af9f-7eb8cb8482b5-kube-api-access-k7l8c\") pod \"payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp\" (UID: \"d22839e6-73a6-412c-af9f-7eb8cb8482b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" Sep 17 23:18:55.888614 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.888539 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" Sep 17 23:18:55.915188 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:55.915160 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" Sep 17 23:18:56.170960 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:56.170932 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp"] Sep 17 23:18:56.172940 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:18:56.172893 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22839e6_73a6_412c_af9f_7eb8cb8482b5.slice/crio-a6862811aa96e8cffe38cc991ec25491e2331f1ce9fead07fdad77821d3bbb2c WatchSource:0}: Error finding container a6862811aa96e8cffe38cc991ec25491e2331f1ce9fead07fdad77821d3bbb2c: Status 404 returned error can't find the container with id a6862811aa96e8cffe38cc991ec25491e2331f1ce9fead07fdad77821d3bbb2c Sep 17 23:18:56.203682 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:56.203662 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49"] Sep 17 23:18:56.207947 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:18:56.207889 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e40474_42f3_497b_9ecd_b75b4b9d1848.slice/crio-2d3c1994ac19bf96a2c67efc57ce2401a6b408674d79d4ce21f371650a2a599a WatchSource:0}: Error finding container 2d3c1994ac19bf96a2c67efc57ce2401a6b408674d79d4ce21f371650a2a599a: Status 404 returned error can't find the container with id 2d3c1994ac19bf96a2c67efc57ce2401a6b408674d79d4ce21f371650a2a599a Sep 17 23:18:56.605713 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:56.605680 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:18:57.088244 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:57.088203 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" event={"ID":"d22839e6-73a6-412c-af9f-7eb8cb8482b5","Type":"ContainerStarted","Data":"86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3"} Sep 17 23:18:57.088447 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:57.088378 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" Sep 17 23:18:57.088447 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:57.088408 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" event={"ID":"d22839e6-73a6-412c-af9f-7eb8cb8482b5","Type":"ContainerStarted","Data":"a6862811aa96e8cffe38cc991ec25491e2331f1ce9fead07fdad77821d3bbb2c"} Sep 17 23:18:57.090019 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:57.089989 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" event={"ID":"39e40474-42f3-497b-9ecd-b75b4b9d1848","Type":"ContainerStarted","Data":"f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63"} Sep 17 23:18:57.090150 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:57.090038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" event={"ID":"39e40474-42f3-497b-9ecd-b75b4b9d1848","Type":"ContainerStarted","Data":"2d3c1994ac19bf96a2c67efc57ce2401a6b408674d79d4ce21f371650a2a599a"} Sep 17 23:18:57.090150 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:57.090085 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" Sep 17 23:18:57.113893 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:57.113852 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" podStartSLOduration=2.113835556 podStartE2EDuration="2.113835556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:57.113832516 +0000 UTC m=+765.231311967" watchObservedRunningTime="2026-09-17 23:18:57.113835556 +0000 UTC m=+765.231315004" Sep 17 23:18:57.137096 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:18:57.137032 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" podStartSLOduration=2.137020747 podStartE2EDuration="2.137020747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:57.135431602 +0000 UTC m=+765.252911053" watchObservedRunningTime="2026-09-17 23:18:57.137020747 +0000 UTC m=+765.254500249" Sep 17 23:19:04.071571 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:04.071537 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" Sep 17 23:19:08.097611 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:08.097578 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" Sep 17 23:19:08.098163 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:08.097667 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" Sep 17 23:19:09.457358 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:09.457329 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv"] Sep 17 23:19:09.465727 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:09.465678 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" Sep 17 23:19:09.468243 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:09.468219 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f38e68-dockercfg-lcv94\"" Sep 17 23:19:09.468378 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:09.468220 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f38e68\"" Sep 17 23:19:09.475611 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:09.475587 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv"] Sep 17 23:19:09.556071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:09.556039 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4v4b\" (UniqueName: \"kubernetes.io/projected/f39bb2dc-7241-4137-aea9-5ef0c6b5a240-kube-api-access-f4v4b\") pod \"payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv\" (UID: \"f39bb2dc-7241-4137-aea9-5ef0c6b5a240\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" Sep 17 23:19:09.556254 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:09.556107 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39bb2dc-7241-4137-aea9-5ef0c6b5a240-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv\" (UID: \"f39bb2dc-7241-4137-aea9-5ef0c6b5a240\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" Sep 17 23:19:09.657139 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:09.657104 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4v4b\" (UniqueName: \"kubernetes.io/projected/f39bb2dc-7241-4137-aea9-5ef0c6b5a240-kube-api-access-f4v4b\") pod \"payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv\" (UID: \"f39bb2dc-7241-4137-aea9-5ef0c6b5a240\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" Sep 17 23:19:09.657312 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:09.657162 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39bb2dc-7241-4137-aea9-5ef0c6b5a240-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv\" (UID: \"f39bb2dc-7241-4137-aea9-5ef0c6b5a240\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" Sep 17 23:19:09.657889 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:09.657870 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39bb2dc-7241-4137-aea9-5ef0c6b5a240-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv\" (UID: \"f39bb2dc-7241-4137-aea9-5ef0c6b5a240\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" Sep 17 23:19:09.664721 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:09.664700 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4v4b\" (UniqueName: \"kubernetes.io/projected/f39bb2dc-7241-4137-aea9-5ef0c6b5a240-kube-api-access-f4v4b\") pod \"payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv\" (UID: \"f39bb2dc-7241-4137-aea9-5ef0c6b5a240\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" Sep 17 23:19:09.779528 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:09.779451 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" Sep 17 23:19:10.050530 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:10.050415 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv"] Sep 17 23:19:10.052774 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:19:10.052748 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39bb2dc_7241_4137_aea9_5ef0c6b5a240.slice/crio-a1f3cbeec30c40dc131613b6084bb2213150dab1a6319093f0a228aa37da04a9 WatchSource:0}: Error finding container a1f3cbeec30c40dc131613b6084bb2213150dab1a6319093f0a228aa37da04a9: Status 404 returned error can't find the container with id a1f3cbeec30c40dc131613b6084bb2213150dab1a6319093f0a228aa37da04a9 Sep 17 23:19:10.164636 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:10.164600 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" event={"ID":"f39bb2dc-7241-4137-aea9-5ef0c6b5a240","Type":"ContainerStarted","Data":"7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1"} Sep 17 23:19:10.164807 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:10.164656 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" event={"ID":"f39bb2dc-7241-4137-aea9-5ef0c6b5a240","Type":"ContainerStarted","Data":"a1f3cbeec30c40dc131613b6084bb2213150dab1a6319093f0a228aa37da04a9"} Sep 17 23:19:10.164807 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:10.164719 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" Sep 17 23:19:10.195610 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:10.195540 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" podStartSLOduration=1.195517648 podStartE2EDuration="1.195517648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:10.193978524 +0000 UTC m=+778.311457997" watchObservedRunningTime="2026-09-17 23:19:10.195517648 +0000 UTC m=+778.312997098" Sep 17 23:19:18.666490 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:18.666458 2559 patch_prober.go:28] interesting pod/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": read tcp 10.132.0.2:45764->10.132.0.48:15021: read: connection reset by peer" start-of-body= Sep 17 23:19:18.666912 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:18.666753 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": read tcp 10.132.0.2:45764->10.132.0.48:15021: read: connection reset by peer" Sep 17 23:19:19.108703 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:19.108667 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:19:19.214431 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:19.214401 2559 generic.go:356] "Generic (PLEG): container finished" podID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerID="a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5" exitCode=0 Sep 17 23:19:19.214600 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:19.214472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" event={"ID":"3ff0c6ea-5409-428a-a22c-4447438235a8","Type":"ContainerDied","Data":"a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5"} Sep 17 23:19:19.214879 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:19.214865 2559 scope.go:117] "RemoveContainer" containerID="a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5" Sep 17 23:19:19.215990 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:19.215950 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:19:19.216057 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:19.216025 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:19:19.216092 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:19.216056 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:19:20.221475 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:20.221444 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" event={"ID":"3ff0c6ea-5409-428a-a22c-4447438235a8","Type":"ContainerStarted","Data":"ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7"} Sep 17 23:19:20.930470 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:20.930432 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:19:20.930659 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:20.930629 2559 patch_prober.go:28] interesting pod/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 17 23:19:20.930725 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:20.930705 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 17 23:19:21.174415 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:21.174387 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" Sep 17 23:19:21.930560 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:21.930526 2559 patch_prober.go:28] interesting pod/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 17 23:19:21.930961 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:21.930576 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 17 23:19:22.930582 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:22.930546 2559 patch_prober.go:28] interesting pod/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 17 23:19:22.930977 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:22.930599 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 17 23:19:23.930763 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:23.930725 2559 patch_prober.go:28] interesting pod/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 17 23:19:23.931182 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:23.930780 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 17 23:19:24.930749 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:24.930712 2559 patch_prober.go:28] interesting pod/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 17 23:19:24.930958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:24.930768 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 17 23:19:25.934515 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:25.934479 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:19:25.935006 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:25.934867 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:19:25.935448 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:25.935428 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:19:30.339131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.339098 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49"] Sep 17 23:19:30.339591 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.339345 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" podUID="39e40474-42f3-497b-9ecd-b75b4b9d1848" containerName="payload-processing" containerID="cri-o://f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63" gracePeriod=30 Sep 17 23:19:30.351173 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.350711 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp"] Sep 17 23:19:30.351173 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.351021 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" podUID="d22839e6-73a6-412c-af9f-7eb8cb8482b5" containerName="payload-pre-processing" containerID="cri-o://86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3" gracePeriod=30 Sep 17 23:19:30.667009 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.665635 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r"] Sep 17 23:19:30.667610 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.665892 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" containerID="cri-o://ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7" gracePeriod=30 Sep 17 23:19:30.688381 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.688338 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" Sep 17 23:19:30.692191 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.692161 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" Sep 17 23:19:30.867623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.867583 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k876s\" (UniqueName: \"kubernetes.io/projected/39e40474-42f3-497b-9ecd-b75b4b9d1848-kube-api-access-k876s\") pod \"39e40474-42f3-497b-9ecd-b75b4b9d1848\" (UID: \"39e40474-42f3-497b-9ecd-b75b4b9d1848\") " Sep 17 23:19:30.867623 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.867629 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22839e6-73a6-412c-af9f-7eb8cb8482b5-plugins-config-volume\") pod \"d22839e6-73a6-412c-af9f-7eb8cb8482b5\" (UID: \"d22839e6-73a6-412c-af9f-7eb8cb8482b5\") " Sep 17 23:19:30.867832 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.867725 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7l8c\" (UniqueName: \"kubernetes.io/projected/d22839e6-73a6-412c-af9f-7eb8cb8482b5-kube-api-access-k7l8c\") pod \"d22839e6-73a6-412c-af9f-7eb8cb8482b5\" (UID: \"d22839e6-73a6-412c-af9f-7eb8cb8482b5\") " Sep 17 23:19:30.867832 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.867786 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e40474-42f3-497b-9ecd-b75b4b9d1848-plugins-config-volume\") pod \"39e40474-42f3-497b-9ecd-b75b4b9d1848\" (UID: \"39e40474-42f3-497b-9ecd-b75b4b9d1848\") " Sep 17 23:19:30.868095 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.868056 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22839e6-73a6-412c-af9f-7eb8cb8482b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d22839e6-73a6-412c-af9f-7eb8cb8482b5" (UID: "d22839e6-73a6-412c-af9f-7eb8cb8482b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:19:30.868226 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.868152 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e40474-42f3-497b-9ecd-b75b4b9d1848-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39e40474-42f3-497b-9ecd-b75b4b9d1848" (UID: "39e40474-42f3-497b-9ecd-b75b4b9d1848"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:19:30.868226 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.868215 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22839e6-73a6-412c-af9f-7eb8cb8482b5-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:30.869774 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.869728 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22839e6-73a6-412c-af9f-7eb8cb8482b5-kube-api-access-k7l8c" (OuterVolumeSpecName: "kube-api-access-k7l8c") pod "d22839e6-73a6-412c-af9f-7eb8cb8482b5" (UID: "d22839e6-73a6-412c-af9f-7eb8cb8482b5"). InnerVolumeSpecName "kube-api-access-k7l8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:19:30.869774 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.869754 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e40474-42f3-497b-9ecd-b75b4b9d1848-kube-api-access-k876s" (OuterVolumeSpecName: "kube-api-access-k876s") pod "39e40474-42f3-497b-9ecd-b75b4b9d1848" (UID: "39e40474-42f3-497b-9ecd-b75b4b9d1848"). InnerVolumeSpecName "kube-api-access-k876s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:19:30.969486 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.969452 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7l8c\" (UniqueName: \"kubernetes.io/projected/d22839e6-73a6-412c-af9f-7eb8cb8482b5-kube-api-access-k7l8c\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:30.969486 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.969481 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e40474-42f3-497b-9ecd-b75b4b9d1848-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:30.969486 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:30.969490 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k876s\" (UniqueName: \"kubernetes.io/projected/39e40474-42f3-497b-9ecd-b75b4b9d1848-kube-api-access-k876s\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:31.294927 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.294879 2559 generic.go:356] "Generic (PLEG): container finished" podID="d22839e6-73a6-412c-af9f-7eb8cb8482b5" containerID="86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3" exitCode=0 Sep 17 23:19:31.295118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.294939 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" Sep 17 23:19:31.295118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.294960 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" event={"ID":"d22839e6-73a6-412c-af9f-7eb8cb8482b5","Type":"ContainerDied","Data":"86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3"} Sep 17 23:19:31.295118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.295006 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp" event={"ID":"d22839e6-73a6-412c-af9f-7eb8cb8482b5","Type":"ContainerDied","Data":"a6862811aa96e8cffe38cc991ec25491e2331f1ce9fead07fdad77821d3bbb2c"} Sep 17 23:19:31.295118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.295036 2559 scope.go:117] "RemoveContainer" containerID="86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3" Sep 17 23:19:31.296771 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.296750 2559 generic.go:356] "Generic (PLEG): container finished" podID="39e40474-42f3-497b-9ecd-b75b4b9d1848" containerID="f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63" exitCode=0 Sep 17 23:19:31.296850 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.296788 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" Sep 17 23:19:31.296850 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.296830 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" event={"ID":"39e40474-42f3-497b-9ecd-b75b4b9d1848","Type":"ContainerDied","Data":"f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63"} Sep 17 23:19:31.296974 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.296867 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49" event={"ID":"39e40474-42f3-497b-9ecd-b75b4b9d1848","Type":"ContainerDied","Data":"2d3c1994ac19bf96a2c67efc57ce2401a6b408674d79d4ce21f371650a2a599a"} Sep 17 23:19:31.305157 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.305141 2559 scope.go:117] "RemoveContainer" containerID="86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3" Sep 17 23:19:31.305462 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:19:31.305437 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3\": container with ID starting with 86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3 not found: ID does not exist" containerID="86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3" Sep 17 23:19:31.305537 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.305474 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3"} err="failed to get container status \"86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3\": rpc error: code = NotFound desc = could not find container \"86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3\": container with ID starting with 86cf83f99b1617fb53babbeb14b864698d488419d141b9e3c5a811c4654036a3 not found: ID does not exist" Sep 17 23:19:31.305537 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.305498 2559 scope.go:117] "RemoveContainer" containerID="f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63" Sep 17 23:19:31.314126 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.314103 2559 scope.go:117] "RemoveContainer" containerID="f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63" Sep 17 23:19:31.314475 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:19:31.314406 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63\": container with ID starting with f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63 not found: ID does not exist" containerID="f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63" Sep 17 23:19:31.314475 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.314437 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63"} err="failed to get container status \"f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63\": rpc error: code = NotFound desc = could not find container \"f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63\": container with ID starting with f5f7d59a84fc080fc171b51186d1f77bda9698e997a18816f917d4f818feec63 not found: ID does not exist" Sep 17 23:19:31.316449 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.316428 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp"] Sep 17 23:19:31.320889 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.320869 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54be2504-7979c6d9d6-6pwmp"] Sep 17 23:19:31.331225 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.331201 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49"] Sep 17 23:19:31.343477 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.343448 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54be2504-5bd7cc4c7-bdq49"] Sep 17 23:19:31.739334 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.739305 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs"] Sep 17 23:19:31.739741 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.739729 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22839e6-73a6-412c-af9f-7eb8cb8482b5" containerName="payload-pre-processing" Sep 17 23:19:31.739781 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.739743 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22839e6-73a6-412c-af9f-7eb8cb8482b5" containerName="payload-pre-processing" Sep 17 23:19:31.739781 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.739778 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e40474-42f3-497b-9ecd-b75b4b9d1848" containerName="payload-processing" Sep 17 23:19:31.739841 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.739783 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e40474-42f3-497b-9ecd-b75b4b9d1848" containerName="payload-processing" Sep 17 23:19:31.739873 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.739849 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22839e6-73a6-412c-af9f-7eb8cb8482b5" containerName="payload-pre-processing" Sep 17 23:19:31.739873 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.739862 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="39e40474-42f3-497b-9ecd-b75b4b9d1848" containerName="payload-processing" Sep 17 23:19:31.744746 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.744716 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.747921 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.747875 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-eb3352f3-gw-openshift-default-dockercfg-6zcb5\"" Sep 17 23:19:31.764805 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.764779 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs"] Sep 17 23:19:31.876522 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.876483 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-token\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.876714 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.876537 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-workload-certs\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.876714 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.876604 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-credential-socket\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.876714 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.876659 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istiod-ca-cert\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.876714 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.876707 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-data\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.876949 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.876735 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-podinfo\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.876949 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.876751 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-workload-socket\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.876949 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.876859 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mh95\" (UniqueName: \"kubernetes.io/projected/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-kube-api-access-8mh95\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.876949 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.876938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-envoy\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978025 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.977985 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mh95\" (UniqueName: \"kubernetes.io/projected/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-kube-api-access-8mh95\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-envoy\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978073 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-token\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978103 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-workload-certs\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-credential-socket\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978145 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istiod-ca-cert\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978180 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-data\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-podinfo\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978576 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-workload-socket\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978633 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978600 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-data\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978787 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978751 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-credential-socket\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978861 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978827 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-workload-certs\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.978936 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978855 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-workload-socket\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.979004 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.978981 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istiod-ca-cert\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.980478 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.980458 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-envoy\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.980737 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.980718 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-podinfo\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.985850 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.985818 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-token\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:31.986023 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:31.986007 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mh95\" (UniqueName: \"kubernetes.io/projected/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-kube-api-access-8mh95\") pod \"e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:32.056094 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:32.056012 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:32.286820 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:32.286783 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39e40474-42f3-497b-9ecd-b75b4b9d1848" path="/var/lib/kubelet/pods/39e40474-42f3-497b-9ecd-b75b4b9d1848/volumes" Sep 17 23:19:32.287273 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:32.287249 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d22839e6-73a6-412c-af9f-7eb8cb8482b5" path="/var/lib/kubelet/pods/d22839e6-73a6-412c-af9f-7eb8cb8482b5/volumes" Sep 17 23:19:32.336020 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:32.335996 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs"] Sep 17 23:19:32.337668 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:19:32.337646 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0c690b_e87c_4c3a_981b_fe8ae6c3110a.slice/crio-14561c4d4d1a31b3e5b5269b773ac21a1ba535ac46da19e16af654b6568b9535 WatchSource:0}: Error finding container 14561c4d4d1a31b3e5b5269b773ac21a1ba535ac46da19e16af654b6568b9535: Status 404 returned error can't find the container with id 14561c4d4d1a31b3e5b5269b773ac21a1ba535ac46da19e16af654b6568b9535 Sep 17 23:19:32.339715 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:32.339683 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:19:32.339773 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:32.339749 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:19:32.339811 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:32.339777 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:19:33.310670 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:33.310627 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" event={"ID":"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a","Type":"ContainerStarted","Data":"f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546"} Sep 17 23:19:33.311079 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:33.310691 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" event={"ID":"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a","Type":"ContainerStarted","Data":"14561c4d4d1a31b3e5b5269b773ac21a1ba535ac46da19e16af654b6568b9535"} Sep 17 23:19:33.330658 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:33.330614 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" podStartSLOduration=2.330599343 podStartE2EDuration="2.330599343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:33.328812278 +0000 UTC m=+801.446291727" watchObservedRunningTime="2026-09-17 23:19:33.330599343 +0000 UTC m=+801.448078793" Sep 17 23:19:34.057061 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:34.057028 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:34.062140 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:34.062118 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:34.315846 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:34.315757 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:34.316782 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:34.316760 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:35.944252 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:35.944220 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:19:36.116032 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.115998 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-credential-socket\") pod \"3ff0c6ea-5409-428a-a22c-4447438235a8\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " Sep 17 23:19:36.116231 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116087 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b64dl\" (UniqueName: \"kubernetes.io/projected/3ff0c6ea-5409-428a-a22c-4447438235a8-kube-api-access-b64dl\") pod \"3ff0c6ea-5409-428a-a22c-4447438235a8\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " Sep 17 23:19:36.116231 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116105 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff0c6ea-5409-428a-a22c-4447438235a8-istiod-ca-cert\") pod \"3ff0c6ea-5409-428a-a22c-4447438235a8\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " Sep 17 23:19:36.116231 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116121 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-workload-socket\") pod \"3ff0c6ea-5409-428a-a22c-4447438235a8\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " Sep 17 23:19:36.116231 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116138 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-data\") pod \"3ff0c6ea-5409-428a-a22c-4447438235a8\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " Sep 17 23:19:36.116231 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116159 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-workload-certs\") pod \"3ff0c6ea-5409-428a-a22c-4447438235a8\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " Sep 17 23:19:36.116231 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116182 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-envoy\") pod \"3ff0c6ea-5409-428a-a22c-4447438235a8\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " Sep 17 23:19:36.116231 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116225 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-token\") pod \"3ff0c6ea-5409-428a-a22c-4447438235a8\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " Sep 17 23:19:36.116595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116265 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-podinfo\") pod \"3ff0c6ea-5409-428a-a22c-4447438235a8\" (UID: \"3ff0c6ea-5409-428a-a22c-4447438235a8\") " Sep 17 23:19:36.116595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116332 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ff0c6ea-5409-428a-a22c-4447438235a8" (UID: "3ff0c6ea-5409-428a-a22c-4447438235a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:19:36.116595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116504 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ff0c6ea-5409-428a-a22c-4447438235a8" (UID: "3ff0c6ea-5409-428a-a22c-4447438235a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:19:36.116595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116514 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff0c6ea-5409-428a-a22c-4447438235a8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ff0c6ea-5409-428a-a22c-4447438235a8" (UID: "3ff0c6ea-5409-428a-a22c-4447438235a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:19:36.116595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116546 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ff0c6ea-5409-428a-a22c-4447438235a8" (UID: "3ff0c6ea-5409-428a-a22c-4447438235a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:19:36.116924 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116652 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:36.116924 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116699 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ff0c6ea-5409-428a-a22c-4447438235a8" (UID: "3ff0c6ea-5409-428a-a22c-4447438235a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:19:36.116924 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116706 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:36.116924 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116723 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff0c6ea-5409-428a-a22c-4447438235a8-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:36.116924 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.116736 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:36.118581 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.118556 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff0c6ea-5409-428a-a22c-4447438235a8-kube-api-access-b64dl" (OuterVolumeSpecName: "kube-api-access-b64dl") pod "3ff0c6ea-5409-428a-a22c-4447438235a8" (UID: "3ff0c6ea-5409-428a-a22c-4447438235a8"). InnerVolumeSpecName "kube-api-access-b64dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:19:36.118581 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.118568 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ff0c6ea-5409-428a-a22c-4447438235a8" (UID: "3ff0c6ea-5409-428a-a22c-4447438235a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:19:36.118754 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.118581 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ff0c6ea-5409-428a-a22c-4447438235a8" (UID: "3ff0c6ea-5409-428a-a22c-4447438235a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:19:36.118754 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.118678 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ff0c6ea-5409-428a-a22c-4447438235a8" (UID: "3ff0c6ea-5409-428a-a22c-4447438235a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:19:36.218055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.218020 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b64dl\" (UniqueName: \"kubernetes.io/projected/3ff0c6ea-5409-428a-a22c-4447438235a8-kube-api-access-b64dl\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:36.218055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.218049 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:36.218055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.218059 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:36.218055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.218066 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:36.218338 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.218073 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff0c6ea-5409-428a-a22c-4447438235a8-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:19:36.327542 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.327509 2559 generic.go:356] "Generic (PLEG): container finished" podID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerID="ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7" exitCode=0 Sep 17 23:19:36.327729 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.327586 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" Sep 17 23:19:36.327729 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.327597 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" event={"ID":"3ff0c6ea-5409-428a-a22c-4447438235a8","Type":"ContainerDied","Data":"ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7"} Sep 17 23:19:36.327729 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.327650 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r" event={"ID":"3ff0c6ea-5409-428a-a22c-4447438235a8","Type":"ContainerDied","Data":"99a621a2405be0a37b2d50ff702acb249e7beb379395a0dd9068cf98f9f72caa"} Sep 17 23:19:36.327729 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.327677 2559 scope.go:117] "RemoveContainer" containerID="ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7" Sep 17 23:19:36.336811 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.336777 2559 scope.go:117] "RemoveContainer" containerID="a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5" Sep 17 23:19:36.344890 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.344864 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r"] Sep 17 23:19:36.347055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.347032 2559 scope.go:117] "RemoveContainer" containerID="ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7" Sep 17 23:19:36.347328 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:19:36.347305 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7\": container with ID starting with ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7 not found: ID does not exist" containerID="ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7" Sep 17 23:19:36.347399 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.347337 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7"} err="failed to get container status \"ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7\": rpc error: code = NotFound desc = could not find container \"ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7\": container with ID starting with ed26dc4cff9c6e6613d7e94989180e6aea848f5c4ef36bb0b404727f93d98cf7 not found: ID does not exist" Sep 17 23:19:36.347399 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.347356 2559 scope.go:117] "RemoveContainer" containerID="a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5" Sep 17 23:19:36.347609 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:19:36.347589 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5\": container with ID starting with a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5 not found: ID does not exist" containerID="a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5" Sep 17 23:19:36.347664 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.347614 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5"} err="failed to get container status \"a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5\": rpc error: code = NotFound desc = could not find container \"a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5\": container with ID starting with a0890acda4f8f621536ddbcb19b49b9cba32aaad27e522b26ff8983971017ee5 not found: ID does not exist" Sep 17 23:19:36.348257 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.348237 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-54be2504-openshift-default-6b896c678b-zvb6r"] Sep 17 23:19:36.585393 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.585354 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r"] Sep 17 23:19:36.585844 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.585826 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" Sep 17 23:19:36.585973 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.585846 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" Sep 17 23:19:36.585973 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.585861 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" Sep 17 23:19:36.585973 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.585869 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" Sep 17 23:19:36.586129 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.586028 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" Sep 17 23:19:36.590516 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.590492 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" Sep 17 23:19:36.593257 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.593232 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-eb3352f3\"" Sep 17 23:19:36.593405 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.593388 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-eb3352f3-dockercfg-bdm7c\"" Sep 17 23:19:36.600534 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.600511 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r"] Sep 17 23:19:36.602869 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.602848 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc"] Sep 17 23:19:36.603499 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.603482 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" containerName="istio-proxy" Sep 17 23:19:36.606762 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.606744 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" Sep 17 23:19:36.620886 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.620864 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc"] Sep 17 23:19:36.722384 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.722305 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42dbef9d-0109-4c40-b171-e65fe51492bc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc\" (UID: \"42dbef9d-0109-4c40-b171-e65fe51492bc\") " pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" Sep 17 23:19:36.722384 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.722341 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faaf320f-5428-4426-9215-46872a0affed-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r\" (UID: \"faaf320f-5428-4426-9215-46872a0affed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" Sep 17 23:19:36.722567 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.722450 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pxq\" (UniqueName: \"kubernetes.io/projected/42dbef9d-0109-4c40-b171-e65fe51492bc-kube-api-access-j5pxq\") pod \"payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc\" (UID: \"42dbef9d-0109-4c40-b171-e65fe51492bc\") " pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" Sep 17 23:19:36.722567 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.722489 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sddtv\" (UniqueName: \"kubernetes.io/projected/faaf320f-5428-4426-9215-46872a0affed-kube-api-access-sddtv\") pod \"payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r\" (UID: \"faaf320f-5428-4426-9215-46872a0affed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" Sep 17 23:19:36.823098 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.823068 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pxq\" (UniqueName: \"kubernetes.io/projected/42dbef9d-0109-4c40-b171-e65fe51492bc-kube-api-access-j5pxq\") pod \"payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc\" (UID: \"42dbef9d-0109-4c40-b171-e65fe51492bc\") " pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" Sep 17 23:19:36.823273 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.823112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sddtv\" (UniqueName: \"kubernetes.io/projected/faaf320f-5428-4426-9215-46872a0affed-kube-api-access-sddtv\") pod \"payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r\" (UID: \"faaf320f-5428-4426-9215-46872a0affed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" Sep 17 23:19:36.823273 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.823146 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42dbef9d-0109-4c40-b171-e65fe51492bc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc\" (UID: \"42dbef9d-0109-4c40-b171-e65fe51492bc\") " pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" Sep 17 23:19:36.823273 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.823164 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faaf320f-5428-4426-9215-46872a0affed-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r\" (UID: \"faaf320f-5428-4426-9215-46872a0affed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" Sep 17 23:19:36.823882 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.823865 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42dbef9d-0109-4c40-b171-e65fe51492bc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc\" (UID: \"42dbef9d-0109-4c40-b171-e65fe51492bc\") " pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" Sep 17 23:19:36.823978 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.823958 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faaf320f-5428-4426-9215-46872a0affed-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r\" (UID: \"faaf320f-5428-4426-9215-46872a0affed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" Sep 17 23:19:36.832738 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.832705 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sddtv\" (UniqueName: \"kubernetes.io/projected/faaf320f-5428-4426-9215-46872a0affed-kube-api-access-sddtv\") pod \"payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r\" (UID: \"faaf320f-5428-4426-9215-46872a0affed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" Sep 17 23:19:36.832872 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.832828 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pxq\" (UniqueName: \"kubernetes.io/projected/42dbef9d-0109-4c40-b171-e65fe51492bc-kube-api-access-j5pxq\") pod \"payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc\" (UID: \"42dbef9d-0109-4c40-b171-e65fe51492bc\") " pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" Sep 17 23:19:36.902580 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.902547 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" Sep 17 23:19:36.919766 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:36.919733 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" Sep 17 23:19:37.183099 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:37.183075 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r"] Sep 17 23:19:37.185288 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:19:37.185264 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaaf320f_5428_4426_9215_46872a0affed.slice/crio-844bc211d37cc6b75b54025b306b5e215392040af18be476f3010f9f50088614 WatchSource:0}: Error finding container 844bc211d37cc6b75b54025b306b5e215392040af18be476f3010f9f50088614: Status 404 returned error can't find the container with id 844bc211d37cc6b75b54025b306b5e215392040af18be476f3010f9f50088614 Sep 17 23:19:37.211032 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:37.211009 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc"] Sep 17 23:19:37.217089 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:19:37.217063 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dbef9d_0109_4c40_b171_e65fe51492bc.slice/crio-e753499fddfa355e25e92b3328d67bcd235ed6e382a4d85d942709725281af05 WatchSource:0}: Error finding container e753499fddfa355e25e92b3328d67bcd235ed6e382a4d85d942709725281af05: Status 404 returned error can't find the container with id e753499fddfa355e25e92b3328d67bcd235ed6e382a4d85d942709725281af05 Sep 17 23:19:37.335604 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:37.335568 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" event={"ID":"42dbef9d-0109-4c40-b171-e65fe51492bc","Type":"ContainerStarted","Data":"1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893"} Sep 17 23:19:37.335756 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:37.335617 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" event={"ID":"42dbef9d-0109-4c40-b171-e65fe51492bc","Type":"ContainerStarted","Data":"e753499fddfa355e25e92b3328d67bcd235ed6e382a4d85d942709725281af05"} Sep 17 23:19:37.335756 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:37.335742 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" Sep 17 23:19:37.337368 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:37.337336 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" event={"ID":"faaf320f-5428-4426-9215-46872a0affed","Type":"ContainerStarted","Data":"b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc"} Sep 17 23:19:37.337491 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:37.337396 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" Sep 17 23:19:37.337491 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:37.337421 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" event={"ID":"faaf320f-5428-4426-9215-46872a0affed","Type":"ContainerStarted","Data":"844bc211d37cc6b75b54025b306b5e215392040af18be476f3010f9f50088614"} Sep 17 23:19:37.354326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:37.354287 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" podStartSLOduration=1.3542735750000001 podStartE2EDuration="1.354273575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:37.352368326 +0000 UTC m=+805.469847786" watchObservedRunningTime="2026-09-17 23:19:37.354273575 +0000 UTC m=+805.471753024" Sep 17 23:19:37.369863 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:37.369821 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" podStartSLOduration=1.369808062 podStartE2EDuration="1.369808062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:37.368222708 +0000 UTC m=+805.485702175" watchObservedRunningTime="2026-09-17 23:19:37.369808062 +0000 UTC m=+805.487287511" Sep 17 23:19:38.102891 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:38.102842 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:19:38.288755 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:38.288721 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ff0c6ea-5409-428a-a22c-4447438235a8" path="/var/lib/kubelet/pods/3ff0c6ea-5409-428a-a22c-4447438235a8/volumes" Sep 17 23:19:48.348093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:48.348059 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" Sep 17 23:19:48.348735 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:48.348711 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" Sep 17 23:19:53.002516 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:53.002477 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:19:53.428401 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:53.428301 2559 generic.go:356] "Generic (PLEG): container finished" podID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerID="f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546" exitCode=0 Sep 17 23:19:53.428401 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:53.428381 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" event={"ID":"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a","Type":"ContainerDied","Data":"f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546"} Sep 17 23:19:53.428851 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:53.428833 2559 scope.go:117] "RemoveContainer" containerID="f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546" Sep 17 23:19:53.429953 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:53.429915 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:19:53.430065 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:53.429991 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:19:53.430065 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:53.430027 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:19:54.435248 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:54.435215 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" event={"ID":"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a","Type":"ContainerStarted","Data":"4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d"} Sep 17 23:19:55.056650 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:55.056591 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:19:55.056890 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:55.056780 2559 patch_prober.go:28] interesting pod/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 17 23:19:55.056890 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:55.056840 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 17 23:19:56.056395 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:56.056348 2559 patch_prober.go:28] interesting pod/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 17 23:19:56.056881 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:56.056428 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 17 23:19:57.056484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:57.056448 2559 patch_prober.go:28] interesting pod/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 17 23:19:57.057114 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:57.057076 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 17 23:19:58.057077 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:58.057040 2559 patch_prober.go:28] interesting pod/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 17 23:19:58.057663 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:58.057628 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 17 23:19:59.056989 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:59.056944 2559 patch_prober.go:28] interesting pod/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 17 23:19:59.057300 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:19:59.057271 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 17 23:20:00.060312 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:00.060282 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:20:00.060697 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:00.060532 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:20:00.061287 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:00.061267 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:20:06.756790 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:06.756755 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc"] Sep 17 23:20:06.757263 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:06.757030 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" podUID="42dbef9d-0109-4c40-b171-e65fe51492bc" containerName="payload-processing" containerID="cri-o://1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893" gracePeriod=30 Sep 17 23:20:06.770918 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:06.770877 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r"] Sep 17 23:20:06.771268 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:06.771222 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" podUID="faaf320f-5428-4426-9215-46872a0affed" containerName="payload-pre-processing" containerID="cri-o://b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc" gracePeriod=30 Sep 17 23:20:07.055516 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.054623 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs"] Sep 17 23:20:07.055516 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.054895 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerName="istio-proxy" containerID="cri-o://4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d" gracePeriod=30 Sep 17 23:20:07.080377 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.080349 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" Sep 17 23:20:07.083568 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.083549 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" Sep 17 23:20:07.190861 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.190829 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faaf320f-5428-4426-9215-46872a0affed-plugins-config-volume\") pod \"faaf320f-5428-4426-9215-46872a0affed\" (UID: \"faaf320f-5428-4426-9215-46872a0affed\") " Sep 17 23:20:07.191069 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.190891 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5pxq\" (UniqueName: \"kubernetes.io/projected/42dbef9d-0109-4c40-b171-e65fe51492bc-kube-api-access-j5pxq\") pod \"42dbef9d-0109-4c40-b171-e65fe51492bc\" (UID: \"42dbef9d-0109-4c40-b171-e65fe51492bc\") " Sep 17 23:20:07.191069 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.190936 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42dbef9d-0109-4c40-b171-e65fe51492bc-plugins-config-volume\") pod \"42dbef9d-0109-4c40-b171-e65fe51492bc\" (UID: \"42dbef9d-0109-4c40-b171-e65fe51492bc\") " Sep 17 23:20:07.191069 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.190968 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sddtv\" (UniqueName: \"kubernetes.io/projected/faaf320f-5428-4426-9215-46872a0affed-kube-api-access-sddtv\") pod \"faaf320f-5428-4426-9215-46872a0affed\" (UID: \"faaf320f-5428-4426-9215-46872a0affed\") " Sep 17 23:20:07.191267 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.191240 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42dbef9d-0109-4c40-b171-e65fe51492bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42dbef9d-0109-4c40-b171-e65fe51492bc" (UID: "42dbef9d-0109-4c40-b171-e65fe51492bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:20:07.191324 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.191303 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faaf320f-5428-4426-9215-46872a0affed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "faaf320f-5428-4426-9215-46872a0affed" (UID: "faaf320f-5428-4426-9215-46872a0affed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:20:07.193014 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.192988 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dbef9d-0109-4c40-b171-e65fe51492bc-kube-api-access-j5pxq" (OuterVolumeSpecName: "kube-api-access-j5pxq") pod "42dbef9d-0109-4c40-b171-e65fe51492bc" (UID: "42dbef9d-0109-4c40-b171-e65fe51492bc"). InnerVolumeSpecName "kube-api-access-j5pxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:20:07.193100 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.193080 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faaf320f-5428-4426-9215-46872a0affed-kube-api-access-sddtv" (OuterVolumeSpecName: "kube-api-access-sddtv") pod "faaf320f-5428-4426-9215-46872a0affed" (UID: "faaf320f-5428-4426-9215-46872a0affed"). InnerVolumeSpecName "kube-api-access-sddtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:20:07.291868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.291841 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5pxq\" (UniqueName: \"kubernetes.io/projected/42dbef9d-0109-4c40-b171-e65fe51492bc-kube-api-access-j5pxq\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:07.291868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.291862 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42dbef9d-0109-4c40-b171-e65fe51492bc-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:07.291868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.291870 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sddtv\" (UniqueName: \"kubernetes.io/projected/faaf320f-5428-4426-9215-46872a0affed-kube-api-access-sddtv\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:07.292064 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.291879 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faaf320f-5428-4426-9215-46872a0affed-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:07.516347 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.516309 2559 generic.go:356] "Generic (PLEG): container finished" podID="42dbef9d-0109-4c40-b171-e65fe51492bc" containerID="1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893" exitCode=0 Sep 17 23:20:07.516347 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.516348 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" Sep 17 23:20:07.516616 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.516396 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" event={"ID":"42dbef9d-0109-4c40-b171-e65fe51492bc","Type":"ContainerDied","Data":"1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893"} Sep 17 23:20:07.516616 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.516445 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc" event={"ID":"42dbef9d-0109-4c40-b171-e65fe51492bc","Type":"ContainerDied","Data":"e753499fddfa355e25e92b3328d67bcd235ed6e382a4d85d942709725281af05"} Sep 17 23:20:07.516616 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.516473 2559 scope.go:117] "RemoveContainer" containerID="1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893" Sep 17 23:20:07.517784 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.517761 2559 generic.go:356] "Generic (PLEG): container finished" podID="faaf320f-5428-4426-9215-46872a0affed" containerID="b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc" exitCode=0 Sep 17 23:20:07.517913 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.517799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" event={"ID":"faaf320f-5428-4426-9215-46872a0affed","Type":"ContainerDied","Data":"b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc"} Sep 17 23:20:07.517913 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.517819 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" Sep 17 23:20:07.517913 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.517845 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r" event={"ID":"faaf320f-5428-4426-9215-46872a0affed","Type":"ContainerDied","Data":"844bc211d37cc6b75b54025b306b5e215392040af18be476f3010f9f50088614"} Sep 17 23:20:07.525160 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.525144 2559 scope.go:117] "RemoveContainer" containerID="1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893" Sep 17 23:20:07.525446 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:20:07.525422 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893\": container with ID starting with 1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893 not found: ID does not exist" containerID="1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893" Sep 17 23:20:07.525514 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.525459 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893"} err="failed to get container status \"1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893\": rpc error: code = NotFound desc = could not find container \"1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893\": container with ID starting with 1e1b5a87270ad7c1d2628a6e43058cd910832221b157cc2cf6d00aca8dde8893 not found: ID does not exist" Sep 17 23:20:07.525514 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.525485 2559 scope.go:117] "RemoveContainer" containerID="b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc" Sep 17 23:20:07.533753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.533737 2559 scope.go:117] "RemoveContainer" containerID="b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc" Sep 17 23:20:07.534004 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:20:07.533988 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc\": container with ID starting with b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc not found: ID does not exist" containerID="b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc" Sep 17 23:20:07.534069 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.534011 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc"} err="failed to get container status \"b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc\": rpc error: code = NotFound desc = could not find container \"b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc\": container with ID starting with b38c5f2cd0048f216655ce5411ba8e137dfb9c995c0b526cbaaa051e809689bc not found: ID does not exist" Sep 17 23:20:07.550163 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.550131 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r"] Sep 17 23:20:07.554460 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.554440 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-eb3352f3-76cfbbb57d-z487r"] Sep 17 23:20:07.565090 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.565069 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc"] Sep 17 23:20:07.569525 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.569505 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-eb3352f3-5fc8458b6b-nvgrc"] Sep 17 23:20:07.781604 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.781530 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh"] Sep 17 23:20:07.782171 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.782154 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faaf320f-5428-4426-9215-46872a0affed" containerName="payload-pre-processing" Sep 17 23:20:07.782224 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.782174 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="faaf320f-5428-4426-9215-46872a0affed" containerName="payload-pre-processing" Sep 17 23:20:07.782224 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.782194 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42dbef9d-0109-4c40-b171-e65fe51492bc" containerName="payload-processing" Sep 17 23:20:07.782224 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.782211 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="42dbef9d-0109-4c40-b171-e65fe51492bc" containerName="payload-processing" Sep 17 23:20:07.782328 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.782317 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="42dbef9d-0109-4c40-b171-e65fe51492bc" containerName="payload-processing" Sep 17 23:20:07.782365 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.782333 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="faaf320f-5428-4426-9215-46872a0affed" containerName="payload-pre-processing" Sep 17 23:20:07.787074 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.787050 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.790193 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.790167 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5ab8e6b6-openshift-default-dockercfg-4rj7j\"" Sep 17 23:20:07.800051 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.800026 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh"] Sep 17 23:20:07.896773 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.896734 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-workload-socket\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.896773 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.896774 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-credential-socket\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.897053 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.896796 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-token\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.897053 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.896987 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-data\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.897053 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.897022 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mh4\" (UniqueName: \"kubernetes.io/projected/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-kube-api-access-x9mh4\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.897220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.897097 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-podinfo\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.897220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.897141 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-workload-certs\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.897220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.897168 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-envoy\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.897220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.897196 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istiod-ca-cert\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998117 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998071 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-data\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998117 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998124 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mh4\" (UniqueName: \"kubernetes.io/projected/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-kube-api-access-x9mh4\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998160 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-podinfo\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-workload-certs\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998214 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-envoy\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998230 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istiod-ca-cert\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998255 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-workload-socket\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-credential-socket\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998307 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-token\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998703 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998490 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-data\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998761 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998732 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-workload-certs\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998816 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998795 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-credential-socket\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.998913 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.998881 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-workload-socket\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:07.999291 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:07.999274 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istiod-ca-cert\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:08.000939 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.000919 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-envoy\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:08.001072 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.001047 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-podinfo\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:08.006702 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.006683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-token\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:08.006997 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.006979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mh4\" (UniqueName: \"kubernetes.io/projected/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-kube-api-access-x9mh4\") pod \"e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:08.101245 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.101175 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:08.288114 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.288072 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42dbef9d-0109-4c40-b171-e65fe51492bc" path="/var/lib/kubelet/pods/42dbef9d-0109-4c40-b171-e65fe51492bc/volumes" Sep 17 23:20:08.288547 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.288527 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faaf320f-5428-4426-9215-46872a0affed" path="/var/lib/kubelet/pods/faaf320f-5428-4426-9215-46872a0affed/volumes" Sep 17 23:20:08.377008 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.376985 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh"] Sep 17 23:20:08.378947 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:20:08.378920 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89b0f7a_ea7a_40b1_82d9_2b414cb72acd.slice/crio-1e30bb467bf8c6ef4fdca867e95420ea05540c5a85dca0793ea28c2981465cbc WatchSource:0}: Error finding container 1e30bb467bf8c6ef4fdca867e95420ea05540c5a85dca0793ea28c2981465cbc: Status 404 returned error can't find the container with id 1e30bb467bf8c6ef4fdca867e95420ea05540c5a85dca0793ea28c2981465cbc Sep 17 23:20:08.381188 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.381158 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:20:08.381261 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.381219 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:20:08.381261 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.381246 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:20:08.526678 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.526642 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" event={"ID":"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd","Type":"ContainerStarted","Data":"63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952"} Sep 17 23:20:08.526829 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.526703 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" event={"ID":"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd","Type":"ContainerStarted","Data":"1e30bb467bf8c6ef4fdca867e95420ea05540c5a85dca0793ea28c2981465cbc"} Sep 17 23:20:08.558529 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:08.558467 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" podStartSLOduration=1.558446076 podStartE2EDuration="1.558446076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:08.557429975 +0000 UTC m=+836.674909425" watchObservedRunningTime="2026-09-17 23:20:08.558446076 +0000 UTC m=+836.675925526" Sep 17 23:20:09.101384 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:09.101353 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:09.107023 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:09.106999 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:09.532034 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:09.532006 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:09.532808 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:09.532788 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:12.326416 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.326352 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:20:12.437298 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437271 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-workload-certs\") pod \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " Sep 17 23:20:12.437470 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437304 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-data\") pod \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " Sep 17 23:20:12.437470 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437345 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-token\") pod \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " Sep 17 23:20:12.437470 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437363 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-podinfo\") pod \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " Sep 17 23:20:12.437470 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437387 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istiod-ca-cert\") pod \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " Sep 17 23:20:12.437470 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437423 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-credential-socket\") pod \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " Sep 17 23:20:12.437470 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437449 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-envoy\") pod \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " Sep 17 23:20:12.437778 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437501 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-workload-socket\") pod \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " Sep 17 23:20:12.437778 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437526 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mh95\" (UniqueName: \"kubernetes.io/projected/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-kube-api-access-8mh95\") pod \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\" (UID: \"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a\") " Sep 17 23:20:12.437778 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437604 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" (UID: "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:20:12.437778 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437618 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" (UID: "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:20:12.437992 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437877 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:12.437992 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437884 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" (UID: "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:20:12.437992 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.437896 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:12.438101 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.438003 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" (UID: "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:20:12.438101 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.438059 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" (UID: "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:20:12.439754 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.439726 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-kube-api-access-8mh95" (OuterVolumeSpecName: "kube-api-access-8mh95") pod "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" (UID: "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a"). InnerVolumeSpecName "kube-api-access-8mh95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:20:12.439889 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.439823 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" (UID: "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:20:12.440050 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.440017 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" (UID: "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:20:12.440153 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.440051 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" (UID: "3d0c690b-e87c-4c3a-981b-fe8ae6c3110a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:20:12.539089 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.539061 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:12.539089 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.539085 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:12.539089 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.539094 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:12.539283 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.539104 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mh95\" (UniqueName: \"kubernetes.io/projected/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-kube-api-access-8mh95\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:12.539283 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.539114 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:12.539283 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.539121 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:12.539283 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.539129 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:12.552999 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.552966 2559 generic.go:356] "Generic (PLEG): container finished" podID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerID="4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d" exitCode=0 Sep 17 23:20:12.553151 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.553022 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" Sep 17 23:20:12.553151 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.553045 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" event={"ID":"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a","Type":"ContainerDied","Data":"4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d"} Sep 17 23:20:12.553151 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.553101 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs" event={"ID":"3d0c690b-e87c-4c3a-981b-fe8ae6c3110a","Type":"ContainerDied","Data":"14561c4d4d1a31b3e5b5269b773ac21a1ba535ac46da19e16af654b6568b9535"} Sep 17 23:20:12.553151 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.553124 2559 scope.go:117] "RemoveContainer" containerID="4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d" Sep 17 23:20:12.564144 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.564125 2559 scope.go:117] "RemoveContainer" containerID="f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546" Sep 17 23:20:12.572319 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.572304 2559 scope.go:117] "RemoveContainer" containerID="4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d" Sep 17 23:20:12.572596 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:20:12.572556 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d\": container with ID starting with 4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d not found: ID does not exist" containerID="4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d" Sep 17 23:20:12.572683 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.572607 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d"} err="failed to get container status \"4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d\": rpc error: code = NotFound desc = could not find container \"4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d\": container with ID starting with 4e45a890e1fd25e0654d3d650ab9cb96c036a438419c0c7c4d10ce0ae3f3dd7d not found: ID does not exist" Sep 17 23:20:12.572683 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.572630 2559 scope.go:117] "RemoveContainer" containerID="f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546" Sep 17 23:20:12.572888 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:20:12.572870 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546\": container with ID starting with f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546 not found: ID does not exist" containerID="f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546" Sep 17 23:20:12.572948 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.572894 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546"} err="failed to get container status \"f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546\": rpc error: code = NotFound desc = could not find container \"f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546\": container with ID starting with f7a9cf19ffe39937082332dda4c79d7c37cde334c353a700aba7df073342d546 not found: ID does not exist" Sep 17 23:20:12.576033 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.576010 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs"] Sep 17 23:20:12.584727 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:12.584676 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-eb3352f3-gw-openshift-default-86dc568c48-qm7hs"] Sep 17 23:20:13.044965 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.044927 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg"] Sep 17 23:20:13.045547 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.045530 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerName="istio-proxy" Sep 17 23:20:13.045637 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.045548 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerName="istio-proxy" Sep 17 23:20:13.045637 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.045579 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerName="istio-proxy" Sep 17 23:20:13.045637 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.045587 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerName="istio-proxy" Sep 17 23:20:13.045789 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.045672 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerName="istio-proxy" Sep 17 23:20:13.045789 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.045684 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" containerName="istio-proxy" Sep 17 23:20:13.050307 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.050288 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" Sep 17 23:20:13.054393 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.054371 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5ab8e6b6\"" Sep 17 23:20:13.054545 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.054526 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5ab8e6b6-dockercfg-bs5fz\"" Sep 17 23:20:13.062818 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.062794 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg"] Sep 17 23:20:13.074120 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.074100 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr"] Sep 17 23:20:13.077566 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.077551 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" Sep 17 23:20:13.090060 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.090039 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr"] Sep 17 23:20:13.144554 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.144519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg\" (UID: \"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" Sep 17 23:20:13.144554 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.144554 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm4p8\" (UniqueName: \"kubernetes.io/projected/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b-kube-api-access-xm4p8\") pod \"payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg\" (UID: \"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" Sep 17 23:20:13.144787 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.144588 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48df99e-b72f-4e3c-bb49-26642afac7a9-plugins-config-volume\") pod \"payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr\" (UID: \"b48df99e-b72f-4e3c-bb49-26642afac7a9\") " pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" Sep 17 23:20:13.144787 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.144702 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbc6\" (UniqueName: \"kubernetes.io/projected/b48df99e-b72f-4e3c-bb49-26642afac7a9-kube-api-access-4mbc6\") pod \"payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr\" (UID: \"b48df99e-b72f-4e3c-bb49-26642afac7a9\") " pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" Sep 17 23:20:13.245619 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.245582 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg\" (UID: \"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" Sep 17 23:20:13.245619 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.245620 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm4p8\" (UniqueName: \"kubernetes.io/projected/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b-kube-api-access-xm4p8\") pod \"payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg\" (UID: \"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" Sep 17 23:20:13.245890 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.245647 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48df99e-b72f-4e3c-bb49-26642afac7a9-plugins-config-volume\") pod \"payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr\" (UID: \"b48df99e-b72f-4e3c-bb49-26642afac7a9\") " pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" Sep 17 23:20:13.245890 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.245724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbc6\" (UniqueName: \"kubernetes.io/projected/b48df99e-b72f-4e3c-bb49-26642afac7a9-kube-api-access-4mbc6\") pod \"payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr\" (UID: \"b48df99e-b72f-4e3c-bb49-26642afac7a9\") " pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" Sep 17 23:20:13.246589 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.246565 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg\" (UID: \"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" Sep 17 23:20:13.246711 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.246567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48df99e-b72f-4e3c-bb49-26642afac7a9-plugins-config-volume\") pod \"payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr\" (UID: \"b48df99e-b72f-4e3c-bb49-26642afac7a9\") " pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" Sep 17 23:20:13.254138 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.254112 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbc6\" (UniqueName: \"kubernetes.io/projected/b48df99e-b72f-4e3c-bb49-26642afac7a9-kube-api-access-4mbc6\") pod \"payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr\" (UID: \"b48df99e-b72f-4e3c-bb49-26642afac7a9\") " pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" Sep 17 23:20:13.254236 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.254198 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm4p8\" (UniqueName: \"kubernetes.io/projected/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b-kube-api-access-xm4p8\") pod \"payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg\" (UID: \"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" Sep 17 23:20:13.361114 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.361035 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" Sep 17 23:20:13.388015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.387988 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" Sep 17 23:20:13.514465 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.514434 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:20:13.635704 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.635651 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg"] Sep 17 23:20:13.638044 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:20:13.638016 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2c7d52_fd9e_484f_a6b5_0a9bae67de2b.slice/crio-ee8c89c34264623fd75cc98fd7b9b218b889bb7cbfcd889db29dad16edb2c07d WatchSource:0}: Error finding container ee8c89c34264623fd75cc98fd7b9b218b889bb7cbfcd889db29dad16edb2c07d: Status 404 returned error can't find the container with id ee8c89c34264623fd75cc98fd7b9b218b889bb7cbfcd889db29dad16edb2c07d Sep 17 23:20:13.667460 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:13.667439 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr"] Sep 17 23:20:13.672988 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:20:13.672966 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48df99e_b72f_4e3c_bb49_26642afac7a9.slice/crio-7bb575229fbfda28b4289933dffaa184b2149662fcdea875beb81864db4af716 WatchSource:0}: Error finding container 7bb575229fbfda28b4289933dffaa184b2149662fcdea875beb81864db4af716: Status 404 returned error can't find the container with id 7bb575229fbfda28b4289933dffaa184b2149662fcdea875beb81864db4af716 Sep 17 23:20:14.287971 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:14.287942 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d0c690b-e87c-4c3a-981b-fe8ae6c3110a" path="/var/lib/kubelet/pods/3d0c690b-e87c-4c3a-981b-fe8ae6c3110a/volumes" Sep 17 23:20:14.567506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:14.567410 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" event={"ID":"b48df99e-b72f-4e3c-bb49-26642afac7a9","Type":"ContainerStarted","Data":"550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4"} Sep 17 23:20:14.567506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:14.567472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" event={"ID":"b48df99e-b72f-4e3c-bb49-26642afac7a9","Type":"ContainerStarted","Data":"7bb575229fbfda28b4289933dffaa184b2149662fcdea875beb81864db4af716"} Sep 17 23:20:14.567949 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:14.567588 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" Sep 17 23:20:14.569078 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:14.569044 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" event={"ID":"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b","Type":"ContainerStarted","Data":"4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb"} Sep 17 23:20:14.569239 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:14.569091 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" event={"ID":"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b","Type":"ContainerStarted","Data":"ee8c89c34264623fd75cc98fd7b9b218b889bb7cbfcd889db29dad16edb2c07d"} Sep 17 23:20:14.569239 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:14.569113 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" Sep 17 23:20:14.595791 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:14.595739 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" podStartSLOduration=1.595724271 podStartE2EDuration="1.595724271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:14.592506187 +0000 UTC m=+842.709985639" watchObservedRunningTime="2026-09-17 23:20:14.595724271 +0000 UTC m=+842.713203715" Sep 17 23:20:14.615266 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:14.615223 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" podStartSLOduration=1.61520938 podStartE2EDuration="1.61520938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:14.613116183 +0000 UTC m=+842.730595685" watchObservedRunningTime="2026-09-17 23:20:14.61520938 +0000 UTC m=+842.732688830" Sep 17 23:20:25.576828 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:25.576795 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" Sep 17 23:20:25.577317 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:25.576873 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" Sep 17 23:20:32.008754 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:32.008713 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:20:33.678941 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:33.678826 2559 generic.go:356] "Generic (PLEG): container finished" podID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerID="63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952" exitCode=0 Sep 17 23:20:33.678941 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:33.678881 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" event={"ID":"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd","Type":"ContainerDied","Data":"63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952"} Sep 17 23:20:33.679422 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:33.679277 2559 scope.go:117] "RemoveContainer" containerID="63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952" Sep 17 23:20:33.680427 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:33.680396 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:20:33.680510 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:33.680461 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:20:33.680510 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:33.680491 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:20:34.687557 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:34.687522 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" event={"ID":"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd","Type":"ContainerStarted","Data":"539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315"} Sep 17 23:20:35.102390 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:35.102354 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:35.103714 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:35.103689 2559 patch_prober.go:28] interesting pod/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 17 23:20:35.103836 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:35.103736 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 17 23:20:36.102146 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:36.102114 2559 patch_prober.go:28] interesting pod/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 17 23:20:36.102522 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:36.102164 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 17 23:20:37.102198 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:37.102157 2559 patch_prober.go:28] interesting pod/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 17 23:20:37.102652 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:37.102223 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 17 23:20:38.101683 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:38.101646 2559 patch_prober.go:28] interesting pod/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 17 23:20:38.101869 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:38.101710 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 17 23:20:39.102292 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:39.102253 2559 patch_prober.go:28] interesting pod/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 17 23:20:39.102758 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:39.102314 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 17 23:20:39.532163 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:39.532133 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:40.105588 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:40.105551 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:40.106535 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:40.106515 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:42.130082 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.130051 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr"] Sep 17 23:20:42.130563 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.130242 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" podUID="b48df99e-b72f-4e3c-bb49-26642afac7a9" containerName="payload-processing" containerID="cri-o://550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4" gracePeriod=30 Sep 17 23:20:42.139756 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.139727 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg"] Sep 17 23:20:42.140174 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.140123 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" podUID="0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b" containerName="payload-pre-processing" containerID="cri-o://4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb" gracePeriod=30 Sep 17 23:20:42.418804 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.418775 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" Sep 17 23:20:42.434618 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.434595 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" Sep 17 23:20:42.522780 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.522744 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mbc6\" (UniqueName: \"kubernetes.io/projected/b48df99e-b72f-4e3c-bb49-26642afac7a9-kube-api-access-4mbc6\") pod \"b48df99e-b72f-4e3c-bb49-26642afac7a9\" (UID: \"b48df99e-b72f-4e3c-bb49-26642afac7a9\") " Sep 17 23:20:42.523002 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.522797 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b-plugins-config-volume\") pod \"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b\" (UID: \"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b\") " Sep 17 23:20:42.523002 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.522875 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48df99e-b72f-4e3c-bb49-26642afac7a9-plugins-config-volume\") pod \"b48df99e-b72f-4e3c-bb49-26642afac7a9\" (UID: \"b48df99e-b72f-4e3c-bb49-26642afac7a9\") " Sep 17 23:20:42.523002 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.522892 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm4p8\" (UniqueName: \"kubernetes.io/projected/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b-kube-api-access-xm4p8\") pod \"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b\" (UID: \"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b\") " Sep 17 23:20:42.523310 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.523280 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48df99e-b72f-4e3c-bb49-26642afac7a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b48df99e-b72f-4e3c-bb49-26642afac7a9" (UID: "b48df99e-b72f-4e3c-bb49-26642afac7a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:20:42.523455 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.523306 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b" (UID: "0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:20:42.525072 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.525051 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48df99e-b72f-4e3c-bb49-26642afac7a9-kube-api-access-4mbc6" (OuterVolumeSpecName: "kube-api-access-4mbc6") pod "b48df99e-b72f-4e3c-bb49-26642afac7a9" (UID: "b48df99e-b72f-4e3c-bb49-26642afac7a9"). InnerVolumeSpecName "kube-api-access-4mbc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:20:42.525148 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.525101 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b-kube-api-access-xm4p8" (OuterVolumeSpecName: "kube-api-access-xm4p8") pod "0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b" (UID: "0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b"). InnerVolumeSpecName "kube-api-access-xm4p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:20:42.623706 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.623675 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48df99e-b72f-4e3c-bb49-26642afac7a9-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:42.623706 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.623705 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm4p8\" (UniqueName: \"kubernetes.io/projected/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b-kube-api-access-xm4p8\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:42.623925 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.623716 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mbc6\" (UniqueName: \"kubernetes.io/projected/b48df99e-b72f-4e3c-bb49-26642afac7a9-kube-api-access-4mbc6\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:42.623925 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.623724 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:42.730317 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.730289 2559 generic.go:356] "Generic (PLEG): container finished" podID="0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b" containerID="4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb" exitCode=0 Sep 17 23:20:42.730505 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.730329 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" Sep 17 23:20:42.730505 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.730378 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" event={"ID":"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b","Type":"ContainerDied","Data":"4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb"} Sep 17 23:20:42.730505 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.730456 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg" event={"ID":"0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b","Type":"ContainerDied","Data":"ee8c89c34264623fd75cc98fd7b9b218b889bb7cbfcd889db29dad16edb2c07d"} Sep 17 23:20:42.730505 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.730485 2559 scope.go:117] "RemoveContainer" containerID="4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb" Sep 17 23:20:42.731850 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.731820 2559 generic.go:356] "Generic (PLEG): container finished" podID="b48df99e-b72f-4e3c-bb49-26642afac7a9" containerID="550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4" exitCode=0 Sep 17 23:20:42.731997 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.731855 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" event={"ID":"b48df99e-b72f-4e3c-bb49-26642afac7a9","Type":"ContainerDied","Data":"550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4"} Sep 17 23:20:42.731997 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.731863 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" Sep 17 23:20:42.731997 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.731916 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr" event={"ID":"b48df99e-b72f-4e3c-bb49-26642afac7a9","Type":"ContainerDied","Data":"7bb575229fbfda28b4289933dffaa184b2149662fcdea875beb81864db4af716"} Sep 17 23:20:42.739934 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.739888 2559 scope.go:117] "RemoveContainer" containerID="4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb" Sep 17 23:20:42.740252 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:20:42.740231 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb\": container with ID starting with 4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb not found: ID does not exist" containerID="4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb" Sep 17 23:20:42.740348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.740260 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb"} err="failed to get container status \"4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb\": rpc error: code = NotFound desc = could not find container \"4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb\": container with ID starting with 4b8d52d367e00f9f918e2092a17dc92a1a9ef824d78e678dc5b67daded8912fb not found: ID does not exist" Sep 17 23:20:42.740348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.740275 2559 scope.go:117] "RemoveContainer" containerID="550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4" Sep 17 23:20:42.748999 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.748984 2559 scope.go:117] "RemoveContainer" containerID="550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4" Sep 17 23:20:42.749264 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:20:42.749240 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4\": container with ID starting with 550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4 not found: ID does not exist" containerID="550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4" Sep 17 23:20:42.749361 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.749262 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4"} err="failed to get container status \"550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4\": rpc error: code = NotFound desc = could not find container \"550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4\": container with ID starting with 550bb2b77c88707daa0ec6dafb4bfbf251ff1b555cc4afee5c34f217651f91b4 not found: ID does not exist" Sep 17 23:20:42.754234 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.754211 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr"] Sep 17 23:20:42.760277 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.760258 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ab8e6b6-7d55d8f5c7-c6wpr"] Sep 17 23:20:42.769931 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.769895 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg"] Sep 17 23:20:42.775381 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:42.775361 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ab8e6b6-fb9bbd6b-d4qdg"] Sep 17 23:20:43.381392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:43.381359 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh"] Sep 17 23:20:43.381853 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:43.381645 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerName="istio-proxy" containerID="cri-o://539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315" gracePeriod=30 Sep 17 23:20:44.124324 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.124292 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m"] Sep 17 23:20:44.124748 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.124735 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b" containerName="payload-pre-processing" Sep 17 23:20:44.124813 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.124750 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b" containerName="payload-pre-processing" Sep 17 23:20:44.124813 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.124762 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48df99e-b72f-4e3c-bb49-26642afac7a9" containerName="payload-processing" Sep 17 23:20:44.124813 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.124768 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48df99e-b72f-4e3c-bb49-26642afac7a9" containerName="payload-processing" Sep 17 23:20:44.125046 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.124837 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48df99e-b72f-4e3c-bb49-26642afac7a9" containerName="payload-processing" Sep 17 23:20:44.125046 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.124848 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b" containerName="payload-pre-processing" Sep 17 23:20:44.127933 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.127913 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.130337 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.130320 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-46ccf8cf-openshift-default-dockercfg-g2kfc\"" Sep 17 23:20:44.137645 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.137620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-podinfo\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.137743 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.137655 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-token\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.137743 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.137673 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrtbf\" (UniqueName: \"kubernetes.io/projected/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-kube-api-access-wrtbf\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.137832 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.137772 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-workload-certs\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.137888 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.137833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-data\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.137959 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.137885 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-envoy\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.138013 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.138002 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istiod-ca-cert\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.138064 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.138049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-credential-socket\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.138110 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.138089 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-workload-socket\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.141789 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.141765 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m"] Sep 17 23:20:44.239128 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239093 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istiod-ca-cert\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.239297 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239150 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-credential-socket\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.239297 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239185 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-workload-socket\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.239297 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239214 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-podinfo\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.239297 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239245 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-token\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.239297 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239269 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtbf\" (UniqueName: \"kubernetes.io/projected/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-kube-api-access-wrtbf\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.239621 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239329 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-workload-certs\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.239621 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239377 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-data\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.239621 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239404 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-envoy\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.239621 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239586 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-credential-socket\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.239621 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239606 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-workload-socket\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.239896 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239874 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-data\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.239974 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-workload-certs\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.240029 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.239969 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istiod-ca-cert\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.242339 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.242279 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-podinfo\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.245643 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.245438 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-envoy\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.250994 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.250974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-token\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.253755 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.253734 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrtbf\" (UniqueName: \"kubernetes.io/projected/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-kube-api-access-wrtbf\") pod \"e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.287629 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.287605 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b" path="/var/lib/kubelet/pods/0c2c7d52-fd9e-484f-a6b5-0a9bae67de2b/volumes" Sep 17 23:20:44.287979 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.287965 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b48df99e-b72f-4e3c-bb49-26642afac7a9" path="/var/lib/kubelet/pods/b48df99e-b72f-4e3c-bb49-26642afac7a9/volumes" Sep 17 23:20:44.437837 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.437755 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:44.722708 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.722687 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m"] Sep 17 23:20:44.724769 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:20:44.724730 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9033b5_e31f_4ec5_839e_245e8be5b4cd.slice/crio-56876785dad55520665280efe356f4063bc41c7603ad5715fcb4e0537bda2b12 WatchSource:0}: Error finding container 56876785dad55520665280efe356f4063bc41c7603ad5715fcb4e0537bda2b12: Status 404 returned error can't find the container with id 56876785dad55520665280efe356f4063bc41c7603ad5715fcb4e0537bda2b12 Sep 17 23:20:44.727053 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.727020 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:20:44.727120 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.727087 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:20:44.727120 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.727116 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:20:44.748221 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:44.748195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" event={"ID":"bb9033b5-e31f-4ec5-839e-245e8be5b4cd","Type":"ContainerStarted","Data":"56876785dad55520665280efe356f4063bc41c7603ad5715fcb4e0537bda2b12"} Sep 17 23:20:45.755090 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:45.755055 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" event={"ID":"bb9033b5-e31f-4ec5-839e-245e8be5b4cd","Type":"ContainerStarted","Data":"af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4"} Sep 17 23:20:45.776720 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:45.776671 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" podStartSLOduration=1.77665687 podStartE2EDuration="1.77665687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:45.774745849 +0000 UTC m=+873.892225324" watchObservedRunningTime="2026-09-17 23:20:45.77665687 +0000 UTC m=+873.894136318" Sep 17 23:20:46.438958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:46.438927 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:46.445480 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:46.445456 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:46.760933 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:46.760890 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:46.761941 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:46.761917 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:20:48.629479 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.629456 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:48.678849 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.678774 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-podinfo\") pod \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " Sep 17 23:20:48.678849 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.678808 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istiod-ca-cert\") pod \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " Sep 17 23:20:48.678849 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.678848 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-envoy\") pod \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " Sep 17 23:20:48.679126 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.678890 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-workload-socket\") pod \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " Sep 17 23:20:48.679126 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.678925 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-token\") pod \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " Sep 17 23:20:48.679126 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.678990 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-data\") pod \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " Sep 17 23:20:48.679126 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.679026 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9mh4\" (UniqueName: \"kubernetes.io/projected/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-kube-api-access-x9mh4\") pod \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " Sep 17 23:20:48.679126 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.679072 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-credential-socket\") pod \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " Sep 17 23:20:48.679126 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.679095 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-workload-certs\") pod \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\" (UID: \"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd\") " Sep 17 23:20:48.679401 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.679187 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" (UID: "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:20:48.679401 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.679303 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" (UID: "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:20:48.679507 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.679482 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:48.679549 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.679503 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:48.679549 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.679381 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" (UID: "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:20:48.679612 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.679554 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" (UID: "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:20:48.679612 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.679570 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-data" (OuterVolumeSpecName: "istio-data") pod "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" (UID: "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:20:48.681372 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.681344 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" (UID: "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:20:48.681518 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.681496 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" (UID: "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:20:48.681607 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.681586 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-token" (OuterVolumeSpecName: "istio-token") pod "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" (UID: "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:20:48.681763 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.681745 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-kube-api-access-x9mh4" (OuterVolumeSpecName: "kube-api-access-x9mh4") pod "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" (UID: "a89b0f7a-ea7a-40b1-82d9-2b414cb72acd"). InnerVolumeSpecName "kube-api-access-x9mh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:20:48.772364 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.772326 2559 generic.go:356] "Generic (PLEG): container finished" podID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerID="539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315" exitCode=0 Sep 17 23:20:48.772508 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.772384 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" Sep 17 23:20:48.772508 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.772411 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" event={"ID":"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd","Type":"ContainerDied","Data":"539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315"} Sep 17 23:20:48.772508 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.772465 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh" event={"ID":"a89b0f7a-ea7a-40b1-82d9-2b414cb72acd","Type":"ContainerDied","Data":"1e30bb467bf8c6ef4fdca867e95420ea05540c5a85dca0793ea28c2981465cbc"} Sep 17 23:20:48.772508 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.772490 2559 scope.go:117] "RemoveContainer" containerID="539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315" Sep 17 23:20:48.780142 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.779996 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:48.780142 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.780016 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:48.780142 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.780029 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:48.780142 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.780038 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:48.780142 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.780046 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9mh4\" (UniqueName: \"kubernetes.io/projected/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-kube-api-access-x9mh4\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:48.780142 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.780060 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:48.780142 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.780073 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:20:48.782205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.782189 2559 scope.go:117] "RemoveContainer" containerID="63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952" Sep 17 23:20:48.790060 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.790044 2559 scope.go:117] "RemoveContainer" containerID="539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315" Sep 17 23:20:48.790300 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:20:48.790282 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315\": container with ID starting with 539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315 not found: ID does not exist" containerID="539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315" Sep 17 23:20:48.790340 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.790307 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315"} err="failed to get container status \"539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315\": rpc error: code = NotFound desc = could not find container \"539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315\": container with ID starting with 539a3921798ceaa4e0e06e48e37f7181a2e24171d1167fd010079709dd8f6315 not found: ID does not exist" Sep 17 23:20:48.790340 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.790324 2559 scope.go:117] "RemoveContainer" containerID="63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952" Sep 17 23:20:48.790580 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:20:48.790562 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952\": container with ID starting with 63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952 not found: ID does not exist" containerID="63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952" Sep 17 23:20:48.790632 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.790586 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952"} err="failed to get container status \"63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952\": rpc error: code = NotFound desc = could not find container \"63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952\": container with ID starting with 63d0ddba146d980acf9a34443ee10d954bd71d91d812f37dbf3ca5d866933952 not found: ID does not exist" Sep 17 23:20:48.795198 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.795179 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh"] Sep 17 23:20:48.799255 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:48.799236 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5ab8e6b6-openshift-default-cfbb585d-r2tlh"] Sep 17 23:20:50.287877 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:50.287845 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" path="/var/lib/kubelet/pods/a89b0f7a-ea7a-40b1-82d9-2b414cb72acd/volumes" Sep 17 23:20:52.175449 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.175407 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9"] Sep 17 23:20:52.175869 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.175856 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerName="istio-proxy" Sep 17 23:20:52.175971 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.175870 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerName="istio-proxy" Sep 17 23:20:52.175971 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.175886 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerName="istio-proxy" Sep 17 23:20:52.175971 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.175891 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerName="istio-proxy" Sep 17 23:20:52.176079 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.175958 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerName="istio-proxy" Sep 17 23:20:52.176079 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.175988 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a89b0f7a-ea7a-40b1-82d9-2b414cb72acd" containerName="istio-proxy" Sep 17 23:20:52.178888 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.178869 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" Sep 17 23:20:52.181330 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.181308 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-46ccf8cf\"" Sep 17 23:20:52.188651 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.188629 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9"] Sep 17 23:20:52.211688 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.211662 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tms5r\" (UniqueName: \"kubernetes.io/projected/976a3e1d-4dd9-4809-82d8-cea8a3150277-kube-api-access-tms5r\") pod \"payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9\" (UID: \"976a3e1d-4dd9-4809-82d8-cea8a3150277\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" Sep 17 23:20:52.211789 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.211703 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976a3e1d-4dd9-4809-82d8-cea8a3150277-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9\" (UID: \"976a3e1d-4dd9-4809-82d8-cea8a3150277\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" Sep 17 23:20:52.212788 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.212764 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g"] Sep 17 23:20:52.215668 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.215651 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" Sep 17 23:20:52.218303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.218285 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-46ccf8cf-dockercfg-h2qm7\"" Sep 17 23:20:52.227670 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.227648 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g"] Sep 17 23:20:52.313045 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.313011 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tms5r\" (UniqueName: \"kubernetes.io/projected/976a3e1d-4dd9-4809-82d8-cea8a3150277-kube-api-access-tms5r\") pod \"payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9\" (UID: \"976a3e1d-4dd9-4809-82d8-cea8a3150277\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" Sep 17 23:20:52.313209 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.313051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976a3e1d-4dd9-4809-82d8-cea8a3150277-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9\" (UID: \"976a3e1d-4dd9-4809-82d8-cea8a3150277\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" Sep 17 23:20:52.313209 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.313131 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdbr\" (UniqueName: \"kubernetes.io/projected/54bdbc3d-9aa6-4420-990c-f25bc2545eeb-kube-api-access-cfdbr\") pod \"payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g\" (UID: \"54bdbc3d-9aa6-4420-990c-f25bc2545eeb\") " pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" Sep 17 23:20:52.313209 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.313167 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdbc3d-9aa6-4420-990c-f25bc2545eeb-plugins-config-volume\") pod \"payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g\" (UID: \"54bdbc3d-9aa6-4420-990c-f25bc2545eeb\") " pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" Sep 17 23:20:52.313847 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.313819 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976a3e1d-4dd9-4809-82d8-cea8a3150277-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9\" (UID: \"976a3e1d-4dd9-4809-82d8-cea8a3150277\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" Sep 17 23:20:52.321849 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.321821 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tms5r\" (UniqueName: \"kubernetes.io/projected/976a3e1d-4dd9-4809-82d8-cea8a3150277-kube-api-access-tms5r\") pod \"payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9\" (UID: \"976a3e1d-4dd9-4809-82d8-cea8a3150277\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" Sep 17 23:20:52.413743 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.413712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdbr\" (UniqueName: \"kubernetes.io/projected/54bdbc3d-9aa6-4420-990c-f25bc2545eeb-kube-api-access-cfdbr\") pod \"payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g\" (UID: \"54bdbc3d-9aa6-4420-990c-f25bc2545eeb\") " pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" Sep 17 23:20:52.413875 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.413756 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdbc3d-9aa6-4420-990c-f25bc2545eeb-plugins-config-volume\") pod \"payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g\" (UID: \"54bdbc3d-9aa6-4420-990c-f25bc2545eeb\") " pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" Sep 17 23:20:52.414491 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.414473 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdbc3d-9aa6-4420-990c-f25bc2545eeb-plugins-config-volume\") pod \"payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g\" (UID: \"54bdbc3d-9aa6-4420-990c-f25bc2545eeb\") " pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" Sep 17 23:20:52.421720 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.421702 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdbr\" (UniqueName: \"kubernetes.io/projected/54bdbc3d-9aa6-4420-990c-f25bc2545eeb-kube-api-access-cfdbr\") pod \"payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g\" (UID: \"54bdbc3d-9aa6-4420-990c-f25bc2545eeb\") " pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" Sep 17 23:20:52.514476 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.514448 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" Sep 17 23:20:52.528674 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.528650 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" Sep 17 23:20:52.787374 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.787350 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9"] Sep 17 23:20:52.789331 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:20:52.789309 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976a3e1d_4dd9_4809_82d8_cea8a3150277.slice/crio-d1b46ea0cbe03ca8254abe677fd18c52def70f09877f9a77f6c6dc4c1898eab0 WatchSource:0}: Error finding container d1b46ea0cbe03ca8254abe677fd18c52def70f09877f9a77f6c6dc4c1898eab0: Status 404 returned error can't find the container with id d1b46ea0cbe03ca8254abe677fd18c52def70f09877f9a77f6c6dc4c1898eab0 Sep 17 23:20:52.797299 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.797270 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" event={"ID":"976a3e1d-4dd9-4809-82d8-cea8a3150277","Type":"ContainerStarted","Data":"d1b46ea0cbe03ca8254abe677fd18c52def70f09877f9a77f6c6dc4c1898eab0"} Sep 17 23:20:52.813305 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:52.813271 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g"] Sep 17 23:20:52.819075 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:20:52.819053 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bdbc3d_9aa6_4420_990c_f25bc2545eeb.slice/crio-220303cd86a2829e8cbe047b78010e9c2674d830fae6c4833e0ff23e7e0bb8b9 WatchSource:0}: Error finding container 220303cd86a2829e8cbe047b78010e9c2674d830fae6c4833e0ff23e7e0bb8b9: Status 404 returned error can't find the container with id 220303cd86a2829e8cbe047b78010e9c2674d830fae6c4833e0ff23e7e0bb8b9 Sep 17 23:20:53.804384 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:53.804347 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" event={"ID":"976a3e1d-4dd9-4809-82d8-cea8a3150277","Type":"ContainerStarted","Data":"8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48"} Sep 17 23:20:53.804847 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:53.804476 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" Sep 17 23:20:53.806032 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:53.806008 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" event={"ID":"54bdbc3d-9aa6-4420-990c-f25bc2545eeb","Type":"ContainerStarted","Data":"054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef"} Sep 17 23:20:53.806149 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:53.806046 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" event={"ID":"54bdbc3d-9aa6-4420-990c-f25bc2545eeb","Type":"ContainerStarted","Data":"220303cd86a2829e8cbe047b78010e9c2674d830fae6c4833e0ff23e7e0bb8b9"} Sep 17 23:20:53.806149 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:53.806080 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" Sep 17 23:20:53.821718 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:53.821672 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" podStartSLOduration=1.8216605110000001 podStartE2EDuration="1.821660511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:53.821294305 +0000 UTC m=+881.938773773" watchObservedRunningTime="2026-09-17 23:20:53.821660511 +0000 UTC m=+881.939139955" Sep 17 23:20:53.843482 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:53.843433 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" podStartSLOduration=1.843416039 podStartE2EDuration="1.843416039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:53.842214265 +0000 UTC m=+881.959693714" watchObservedRunningTime="2026-09-17 23:20:53.843416039 +0000 UTC m=+881.960895490" Sep 17 23:20:57.502859 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:20:57.502778 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:21:04.813292 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:04.813256 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" Sep 17 23:21:04.813726 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:04.813403 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" Sep 17 23:21:07.892947 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:07.892914 2559 generic.go:356] "Generic (PLEG): container finished" podID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerID="af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4" exitCode=0 Sep 17 23:21:07.892947 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:07.892927 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" event={"ID":"bb9033b5-e31f-4ec5-839e-245e8be5b4cd","Type":"ContainerDied","Data":"af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4"} Sep 17 23:21:07.893394 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:07.893345 2559 scope.go:117] "RemoveContainer" containerID="af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4" Sep 17 23:21:07.894442 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:07.894413 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:21:07.894539 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:07.894481 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:21:07.894539 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:07.894512 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:21:08.900475 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:08.900443 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" event={"ID":"bb9033b5-e31f-4ec5-839e-245e8be5b4cd","Type":"ContainerStarted","Data":"deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc"} Sep 17 23:21:09.265678 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.265636 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms"] Sep 17 23:21:09.269454 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.269421 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms" Sep 17 23:21:09.272160 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.272133 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 23:21:09.272356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.272294 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d9szq\"" Sep 17 23:21:09.274146 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.274121 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms"] Sep 17 23:21:09.365272 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.365218 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/118ee1a1-5435-446f-9584-1207c2369af7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-61e623-dc9ms\" (UID: \"118ee1a1-5435-446f-9584-1207c2369af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms" Sep 17 23:21:09.438971 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.438942 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:21:09.439155 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.439137 2559 patch_prober.go:28] interesting pod/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 17 23:21:09.439209 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.439188 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 17 23:21:09.466595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.466550 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/118ee1a1-5435-446f-9584-1207c2369af7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-61e623-dc9ms\" (UID: \"118ee1a1-5435-446f-9584-1207c2369af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms" Sep 17 23:21:09.467391 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.467363 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/118ee1a1-5435-446f-9584-1207c2369af7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-61e623-dc9ms\" (UID: \"118ee1a1-5435-446f-9584-1207c2369af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms" Sep 17 23:21:09.582130 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.582039 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms" Sep 17 23:21:09.856276 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.853579 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms"] Sep 17 23:21:09.856629 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:21:09.856587 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118ee1a1_5435_446f_9584_1207c2369af7.slice/crio-51a483e80543fe4b6e6a037b536a79beeeb9969df4272623ec7905736d83f6c7 WatchSource:0}: Error finding container 51a483e80543fe4b6e6a037b536a79beeeb9969df4272623ec7905736d83f6c7: Status 404 returned error can't find the container with id 51a483e80543fe4b6e6a037b536a79beeeb9969df4272623ec7905736d83f6c7 Sep 17 23:21:09.907682 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:09.907653 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms" event={"ID":"118ee1a1-5435-446f-9584-1207c2369af7","Type":"ContainerStarted","Data":"51a483e80543fe4b6e6a037b536a79beeeb9969df4272623ec7905736d83f6c7"} Sep 17 23:21:10.438962 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:10.438926 2559 patch_prober.go:28] interesting pod/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 17 23:21:10.439139 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:10.439016 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 17 23:21:11.438330 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:11.438299 2559 patch_prober.go:28] interesting pod/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 17 23:21:11.438791 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:11.438355 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 17 23:21:12.438806 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:12.438769 2559 patch_prober.go:28] interesting pod/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 17 23:21:12.439181 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:12.438833 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 17 23:21:13.439055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:13.438975 2559 patch_prober.go:28] interesting pod/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 17 23:21:13.439055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:13.439037 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 17 23:21:13.936659 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:13.936626 2559 generic.go:356] "Generic (PLEG): container finished" podID="118ee1a1-5435-446f-9584-1207c2369af7" containerID="2ba9cec8b0d888cb30d6b38a19a844957e69f329b26b8ad3cca4b42b89be0e82" exitCode=0 Sep 17 23:21:13.936848 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:13.936687 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms" event={"ID":"118ee1a1-5435-446f-9584-1207c2369af7","Type":"ContainerDied","Data":"2ba9cec8b0d888cb30d6b38a19a844957e69f329b26b8ad3cca4b42b89be0e82"} Sep 17 23:21:14.222941 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:14.222912 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:21:14.224654 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:14.224634 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:21:14.443222 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:14.443190 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:21:14.443678 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:14.443658 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:21:14.444520 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:14.444486 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:21:15.090851 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:15.090829 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms" Sep 17 23:21:15.227202 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:15.227173 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/118ee1a1-5435-446f-9584-1207c2369af7-maas-api-service-ca\") pod \"118ee1a1-5435-446f-9584-1207c2369af7\" (UID: \"118ee1a1-5435-446f-9584-1207c2369af7\") " Sep 17 23:21:15.227512 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:15.227489 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118ee1a1-5435-446f-9584-1207c2369af7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "118ee1a1-5435-446f-9584-1207c2369af7" (UID: "118ee1a1-5435-446f-9584-1207c2369af7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:21:15.328105 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:15.328076 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/118ee1a1-5435-446f-9584-1207c2369af7-maas-api-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:15.952704 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:15.952662 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms" event={"ID":"118ee1a1-5435-446f-9584-1207c2369af7","Type":"ContainerDied","Data":"51a483e80543fe4b6e6a037b536a79beeeb9969df4272623ec7905736d83f6c7"} Sep 17 23:21:15.952704 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:15.952694 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms" Sep 17 23:21:15.953139 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:15.952725 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a483e80543fe4b6e6a037b536a79beeeb9969df4272623ec7905736d83f6c7" Sep 17 23:21:21.438665 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.438633 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4"] Sep 17 23:21:21.439156 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.438821 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" podUID="0745ffc0-3e9a-443d-94f0-ae7a3f72e550" containerName="payload-processing" containerID="cri-o://89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26" gracePeriod=30 Sep 17 23:21:21.618564 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.618530 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms"] Sep 17 23:21:21.621960 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.621933 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-61e623-dc9ms"] Sep 17 23:21:21.700012 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.699990 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" Sep 17 23:21:21.885314 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.885283 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0745ffc0-3e9a-443d-94f0-ae7a3f72e550-plugins-config-volume\") pod \"0745ffc0-3e9a-443d-94f0-ae7a3f72e550\" (UID: \"0745ffc0-3e9a-443d-94f0-ae7a3f72e550\") " Sep 17 23:21:21.885511 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.885329 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpbm5\" (UniqueName: \"kubernetes.io/projected/0745ffc0-3e9a-443d-94f0-ae7a3f72e550-kube-api-access-vpbm5\") pod \"0745ffc0-3e9a-443d-94f0-ae7a3f72e550\" (UID: \"0745ffc0-3e9a-443d-94f0-ae7a3f72e550\") " Sep 17 23:21:21.885666 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.885643 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0745ffc0-3e9a-443d-94f0-ae7a3f72e550-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0745ffc0-3e9a-443d-94f0-ae7a3f72e550" (UID: "0745ffc0-3e9a-443d-94f0-ae7a3f72e550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:21:21.887428 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.887408 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0745ffc0-3e9a-443d-94f0-ae7a3f72e550-kube-api-access-vpbm5" (OuterVolumeSpecName: "kube-api-access-vpbm5") pod "0745ffc0-3e9a-443d-94f0-ae7a3f72e550" (UID: "0745ffc0-3e9a-443d-94f0-ae7a3f72e550"). InnerVolumeSpecName "kube-api-access-vpbm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:21:21.987008 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.986940 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0745ffc0-3e9a-443d-94f0-ae7a3f72e550-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:21.987008 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.986963 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpbm5\" (UniqueName: \"kubernetes.io/projected/0745ffc0-3e9a-443d-94f0-ae7a3f72e550-kube-api-access-vpbm5\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:21.992538 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.992506 2559 generic.go:356] "Generic (PLEG): container finished" podID="0745ffc0-3e9a-443d-94f0-ae7a3f72e550" containerID="89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26" exitCode=0 Sep 17 23:21:21.992669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.992544 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" Sep 17 23:21:21.992669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.992589 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" event={"ID":"0745ffc0-3e9a-443d-94f0-ae7a3f72e550","Type":"ContainerDied","Data":"89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26"} Sep 17 23:21:21.992669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.992641 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4" event={"ID":"0745ffc0-3e9a-443d-94f0-ae7a3f72e550","Type":"ContainerDied","Data":"ecf16f48ed8da4ef0177ab56108327f4827cd25c24480c77fbb588c1f2057c20"} Sep 17 23:21:21.992669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:21.992663 2559 scope.go:117] "RemoveContainer" containerID="89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26" Sep 17 23:21:22.002254 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:22.002235 2559 scope.go:117] "RemoveContainer" containerID="89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26" Sep 17 23:21:22.002545 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:21:22.002526 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26\": container with ID starting with 89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26 not found: ID does not exist" containerID="89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26" Sep 17 23:21:22.002597 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:22.002555 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26"} err="failed to get container status \"89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26\": rpc error: code = NotFound desc = could not find container \"89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26\": container with ID starting with 89f71e85edeebd475cb3375c2ea0ada1a3484c4425eeba5b23c8172e939e4e26 not found: ID does not exist" Sep 17 23:21:22.017598 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:22.017570 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4"] Sep 17 23:21:22.022620 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:22.022597 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-61e623-54fc96f859-lxvv4"] Sep 17 23:21:22.287827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:22.287750 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0745ffc0-3e9a-443d-94f0-ae7a3f72e550" path="/var/lib/kubelet/pods/0745ffc0-3e9a-443d-94f0-ae7a3f72e550/volumes" Sep 17 23:21:22.288135 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:22.288120 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="118ee1a1-5435-446f-9584-1207c2369af7" path="/var/lib/kubelet/pods/118ee1a1-5435-446f-9584-1207c2369af7/volumes" Sep 17 23:21:23.403350 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:23.403318 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:21:29.250457 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.250421 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g"] Sep 17 23:21:29.250845 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.250699 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" podUID="54bdbc3d-9aa6-4420-990c-f25bc2545eeb" containerName="payload-processing" containerID="cri-o://054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef" gracePeriod=30 Sep 17 23:21:29.257103 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.257075 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9"] Sep 17 23:21:29.257319 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.257295 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" podUID="976a3e1d-4dd9-4809-82d8-cea8a3150277" containerName="payload-pre-processing" containerID="cri-o://8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48" gracePeriod=30 Sep 17 23:21:29.499392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.499369 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" Sep 17 23:21:29.544657 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.544132 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" Sep 17 23:21:29.551400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.550073 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m"] Sep 17 23:21:29.551400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.550282 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerName="istio-proxy" containerID="cri-o://deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc" gracePeriod=30 Sep 17 23:21:29.551400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.550315 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tms5r\" (UniqueName: \"kubernetes.io/projected/976a3e1d-4dd9-4809-82d8-cea8a3150277-kube-api-access-tms5r\") pod \"976a3e1d-4dd9-4809-82d8-cea8a3150277\" (UID: \"976a3e1d-4dd9-4809-82d8-cea8a3150277\") " Sep 17 23:21:29.551400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.550365 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976a3e1d-4dd9-4809-82d8-cea8a3150277-plugins-config-volume\") pod \"976a3e1d-4dd9-4809-82d8-cea8a3150277\" (UID: \"976a3e1d-4dd9-4809-82d8-cea8a3150277\") " Sep 17 23:21:29.551400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.550394 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdbc3d-9aa6-4420-990c-f25bc2545eeb-plugins-config-volume\") pod \"54bdbc3d-9aa6-4420-990c-f25bc2545eeb\" (UID: \"54bdbc3d-9aa6-4420-990c-f25bc2545eeb\") " Sep 17 23:21:29.551400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.550460 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfdbr\" (UniqueName: \"kubernetes.io/projected/54bdbc3d-9aa6-4420-990c-f25bc2545eeb-kube-api-access-cfdbr\") pod \"54bdbc3d-9aa6-4420-990c-f25bc2545eeb\" (UID: \"54bdbc3d-9aa6-4420-990c-f25bc2545eeb\") " Sep 17 23:21:29.551400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.551339 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bdbc3d-9aa6-4420-990c-f25bc2545eeb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54bdbc3d-9aa6-4420-990c-f25bc2545eeb" (UID: "54bdbc3d-9aa6-4420-990c-f25bc2545eeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:21:29.551914 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.551754 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976a3e1d-4dd9-4809-82d8-cea8a3150277-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "976a3e1d-4dd9-4809-82d8-cea8a3150277" (UID: "976a3e1d-4dd9-4809-82d8-cea8a3150277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:21:29.553135 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.553097 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bdbc3d-9aa6-4420-990c-f25bc2545eeb-kube-api-access-cfdbr" (OuterVolumeSpecName: "kube-api-access-cfdbr") pod "54bdbc3d-9aa6-4420-990c-f25bc2545eeb" (UID: "54bdbc3d-9aa6-4420-990c-f25bc2545eeb"). InnerVolumeSpecName "kube-api-access-cfdbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:21:29.554773 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.554746 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976a3e1d-4dd9-4809-82d8-cea8a3150277-kube-api-access-tms5r" (OuterVolumeSpecName: "kube-api-access-tms5r") pod "976a3e1d-4dd9-4809-82d8-cea8a3150277" (UID: "976a3e1d-4dd9-4809-82d8-cea8a3150277"). InnerVolumeSpecName "kube-api-access-tms5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:21:29.651172 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.651139 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tms5r\" (UniqueName: \"kubernetes.io/projected/976a3e1d-4dd9-4809-82d8-cea8a3150277-kube-api-access-tms5r\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:29.651172 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.651168 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976a3e1d-4dd9-4809-82d8-cea8a3150277-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:29.651363 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.651181 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdbc3d-9aa6-4420-990c-f25bc2545eeb-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:29.651363 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:29.651195 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfdbr\" (UniqueName: \"kubernetes.io/projected/54bdbc3d-9aa6-4420-990c-f25bc2545eeb-kube-api-access-cfdbr\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:30.037379 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.037345 2559 generic.go:356] "Generic (PLEG): container finished" podID="54bdbc3d-9aa6-4420-990c-f25bc2545eeb" containerID="054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef" exitCode=0 Sep 17 23:21:30.037556 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.037384 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" Sep 17 23:21:30.037556 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.037427 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" event={"ID":"54bdbc3d-9aa6-4420-990c-f25bc2545eeb","Type":"ContainerDied","Data":"054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef"} Sep 17 23:21:30.037556 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.037478 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g" event={"ID":"54bdbc3d-9aa6-4420-990c-f25bc2545eeb","Type":"ContainerDied","Data":"220303cd86a2829e8cbe047b78010e9c2674d830fae6c4833e0ff23e7e0bb8b9"} Sep 17 23:21:30.037556 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.037502 2559 scope.go:117] "RemoveContainer" containerID="054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef" Sep 17 23:21:30.038968 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.038950 2559 generic.go:356] "Generic (PLEG): container finished" podID="976a3e1d-4dd9-4809-82d8-cea8a3150277" containerID="8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48" exitCode=0 Sep 17 23:21:30.039063 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.038977 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" event={"ID":"976a3e1d-4dd9-4809-82d8-cea8a3150277","Type":"ContainerDied","Data":"8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48"} Sep 17 23:21:30.039063 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.039013 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" event={"ID":"976a3e1d-4dd9-4809-82d8-cea8a3150277","Type":"ContainerDied","Data":"d1b46ea0cbe03ca8254abe677fd18c52def70f09877f9a77f6c6dc4c1898eab0"} Sep 17 23:21:30.039063 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.039040 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9" Sep 17 23:21:30.049746 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.049645 2559 scope.go:117] "RemoveContainer" containerID="054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef" Sep 17 23:21:30.049980 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:21:30.049955 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef\": container with ID starting with 054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef not found: ID does not exist" containerID="054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef" Sep 17 23:21:30.050076 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.049991 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef"} err="failed to get container status \"054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef\": rpc error: code = NotFound desc = could not find container \"054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef\": container with ID starting with 054bbedd799c7d9bf2c0ef756fe4940178affca66a693a86515ecd12a18f08ef not found: ID does not exist" Sep 17 23:21:30.050076 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.050013 2559 scope.go:117] "RemoveContainer" containerID="8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48" Sep 17 23:21:30.058065 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.058048 2559 scope.go:117] "RemoveContainer" containerID="8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48" Sep 17 23:21:30.058319 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:21:30.058303 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48\": container with ID starting with 8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48 not found: ID does not exist" containerID="8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48" Sep 17 23:21:30.058360 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.058326 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48"} err="failed to get container status \"8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48\": rpc error: code = NotFound desc = could not find container \"8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48\": container with ID starting with 8b9ddd5ef42dee23f24f0c1621a50bb458821baa50df06ac52755ca110ffbd48 not found: ID does not exist" Sep 17 23:21:30.063485 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.063461 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9"] Sep 17 23:21:30.068155 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.068132 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46ccf8cf-94944b954-4bbb9"] Sep 17 23:21:30.078125 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.078106 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g"] Sep 17 23:21:30.083742 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.083724 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46ccf8cf-75bd986b65-qkb4g"] Sep 17 23:21:30.293542 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.293471 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54bdbc3d-9aa6-4420-990c-f25bc2545eeb" path="/var/lib/kubelet/pods/54bdbc3d-9aa6-4420-990c-f25bc2545eeb/volumes" Sep 17 23:21:30.293854 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:30.293816 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976a3e1d-4dd9-4809-82d8-cea8a3150277" path="/var/lib/kubelet/pods/976a3e1d-4dd9-4809-82d8-cea8a3150277/volumes" Sep 17 23:21:34.808218 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.808190 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:21:34.895257 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895176 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-data\") pod \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " Sep 17 23:21:34.895257 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895218 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-envoy\") pod \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " Sep 17 23:21:34.895257 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895235 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istiod-ca-cert\") pod \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " Sep 17 23:21:34.895257 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895262 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-workload-socket\") pod \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " Sep 17 23:21:34.895588 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895310 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-credential-socket\") pod \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " Sep 17 23:21:34.895588 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895353 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrtbf\" (UniqueName: \"kubernetes.io/projected/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-kube-api-access-wrtbf\") pod \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " Sep 17 23:21:34.895588 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895394 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-podinfo\") pod \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " Sep 17 23:21:34.895588 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895408 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-token\") pod \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " Sep 17 23:21:34.895588 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895440 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-workload-certs\") pod \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\" (UID: \"bb9033b5-e31f-4ec5-839e-245e8be5b4cd\") " Sep 17 23:21:34.895588 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895488 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-data" (OuterVolumeSpecName: "istio-data") pod "bb9033b5-e31f-4ec5-839e-245e8be5b4cd" (UID: "bb9033b5-e31f-4ec5-839e-245e8be5b4cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:21:34.895879 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895601 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bb9033b5-e31f-4ec5-839e-245e8be5b4cd" (UID: "bb9033b5-e31f-4ec5-839e-245e8be5b4cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:21:34.895879 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895661 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bb9033b5-e31f-4ec5-839e-245e8be5b4cd" (UID: "bb9033b5-e31f-4ec5-839e-245e8be5b4cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:21:34.895879 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895674 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bb9033b5-e31f-4ec5-839e-245e8be5b4cd" (UID: "bb9033b5-e31f-4ec5-839e-245e8be5b4cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:21:34.895879 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895780 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:34.895879 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895799 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:34.895879 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895815 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:34.895879 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895828 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:34.896159 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.895947 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bb9033b5-e31f-4ec5-839e-245e8be5b4cd" (UID: "bb9033b5-e31f-4ec5-839e-245e8be5b4cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:21:34.897314 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.897295 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bb9033b5-e31f-4ec5-839e-245e8be5b4cd" (UID: "bb9033b5-e31f-4ec5-839e-245e8be5b4cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:21:34.897468 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.897453 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb9033b5-e31f-4ec5-839e-245e8be5b4cd" (UID: "bb9033b5-e31f-4ec5-839e-245e8be5b4cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:21:34.897815 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.897800 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bb9033b5-e31f-4ec5-839e-245e8be5b4cd" (UID: "bb9033b5-e31f-4ec5-839e-245e8be5b4cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:21:34.897877 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.897826 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-kube-api-access-wrtbf" (OuterVolumeSpecName: "kube-api-access-wrtbf") pod "bb9033b5-e31f-4ec5-839e-245e8be5b4cd" (UID: "bb9033b5-e31f-4ec5-839e-245e8be5b4cd"). InnerVolumeSpecName "kube-api-access-wrtbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:21:34.996424 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.996400 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrtbf\" (UniqueName: \"kubernetes.io/projected/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-kube-api-access-wrtbf\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:34.996424 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.996420 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:34.996590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.996432 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:34.996590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.996440 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:34.996590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:34.996448 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9033b5-e31f-4ec5-839e-245e8be5b4cd-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:21:35.069684 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:35.069654 2559 generic.go:356] "Generic (PLEG): container finished" podID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerID="deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc" exitCode=0 Sep 17 23:21:35.069820 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:35.069698 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" event={"ID":"bb9033b5-e31f-4ec5-839e-245e8be5b4cd","Type":"ContainerDied","Data":"deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc"} Sep 17 23:21:35.069820 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:35.069702 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" Sep 17 23:21:35.069820 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:35.069739 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m" event={"ID":"bb9033b5-e31f-4ec5-839e-245e8be5b4cd","Type":"ContainerDied","Data":"56876785dad55520665280efe356f4063bc41c7603ad5715fcb4e0537bda2b12"} Sep 17 23:21:35.069820 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:35.069760 2559 scope.go:117] "RemoveContainer" containerID="deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc" Sep 17 23:21:35.078690 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:35.078672 2559 scope.go:117] "RemoveContainer" containerID="af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4" Sep 17 23:21:35.086531 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:35.086515 2559 scope.go:117] "RemoveContainer" containerID="deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc" Sep 17 23:21:35.086766 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:21:35.086750 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc\": container with ID starting with deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc not found: ID does not exist" containerID="deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc" Sep 17 23:21:35.086806 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:35.086772 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc"} err="failed to get container status \"deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc\": rpc error: code = NotFound desc = could not find container \"deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc\": container with ID starting with deff19ec3c60fb1c52e5881c3ecfcd1480af546795cfa0391b6e56134b953cbc not found: ID does not exist" Sep 17 23:21:35.086806 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:35.086789 2559 scope.go:117] "RemoveContainer" containerID="af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4" Sep 17 23:21:35.087059 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:21:35.087044 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4\": container with ID starting with af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4 not found: ID does not exist" containerID="af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4" Sep 17 23:21:35.087118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:35.087061 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4"} err="failed to get container status \"af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4\": rpc error: code = NotFound desc = could not find container \"af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4\": container with ID starting with af20a46236c3bd0b535341113ef659e61345ccd479f555332aacffd166c9f2b4 not found: ID does not exist" Sep 17 23:21:35.091135 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:35.091114 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m"] Sep 17 23:21:35.095105 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:35.095086 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-46ccf8cf-openshift-default-6859757964-2q88m"] Sep 17 23:21:36.287280 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:36.287251 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" path="/var/lib/kubelet/pods/bb9033b5-e31f-4ec5-839e-245e8be5b4cd/volumes" Sep 17 23:21:38.211633 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:21:38.211596 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:22:00.702892 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:00.702855 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:22:05.078087 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078054 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz"] Sep 17 23:22:05.078655 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078639 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976a3e1d-4dd9-4809-82d8-cea8a3150277" containerName="payload-pre-processing" Sep 17 23:22:05.078712 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078658 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="976a3e1d-4dd9-4809-82d8-cea8a3150277" containerName="payload-pre-processing" Sep 17 23:22:05.078712 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078677 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54bdbc3d-9aa6-4420-990c-f25bc2545eeb" containerName="payload-processing" Sep 17 23:22:05.078712 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078687 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="54bdbc3d-9aa6-4420-990c-f25bc2545eeb" containerName="payload-processing" Sep 17 23:22:05.078712 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078707 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0745ffc0-3e9a-443d-94f0-ae7a3f72e550" containerName="payload-processing" Sep 17 23:22:05.078856 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078715 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0745ffc0-3e9a-443d-94f0-ae7a3f72e550" containerName="payload-processing" Sep 17 23:22:05.078856 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078725 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerName="istio-proxy" Sep 17 23:22:05.078856 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078733 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerName="istio-proxy" Sep 17 23:22:05.078856 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078741 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerName="istio-proxy" Sep 17 23:22:05.078856 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078749 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerName="istio-proxy" Sep 17 23:22:05.078856 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078757 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118ee1a1-5435-446f-9584-1207c2369af7" containerName="revoke-keys" Sep 17 23:22:05.078856 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078765 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="118ee1a1-5435-446f-9584-1207c2369af7" containerName="revoke-keys" Sep 17 23:22:05.079111 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078872 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="54bdbc3d-9aa6-4420-990c-f25bc2545eeb" containerName="payload-processing" Sep 17 23:22:05.079111 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078882 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="118ee1a1-5435-446f-9584-1207c2369af7" containerName="revoke-keys" Sep 17 23:22:05.079111 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078915 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="976a3e1d-4dd9-4809-82d8-cea8a3150277" containerName="payload-pre-processing" Sep 17 23:22:05.079111 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078925 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0745ffc0-3e9a-443d-94f0-ae7a3f72e550" containerName="payload-processing" Sep 17 23:22:05.079111 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078934 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerName="istio-proxy" Sep 17 23:22:05.079111 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.078942 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb9033b5-e31f-4ec5-839e-245e8be5b4cd" containerName="istio-proxy" Sep 17 23:22:05.081601 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.081583 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.083975 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.083955 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 23:22:05.084090 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.083991 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-35df6d-openshift-default-dockercfg-m4d4w\"" Sep 17 23:22:05.093539 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.093512 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz"] Sep 17 23:22:05.244429 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.244390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-workload-socket\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.244636 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.244433 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-envoy\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.244636 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.244473 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-token\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.244636 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.244514 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-credential-socket\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.244636 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.244545 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-data\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.244636 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.244599 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d40266b-e47c-4c17-8245-e25b227aa17a-istiod-ca-cert\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.244636 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.244630 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqv8r\" (UniqueName: \"kubernetes.io/projected/3d40266b-e47c-4c17-8245-e25b227aa17a-kube-api-access-dqv8r\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.244981 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.244660 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-podinfo\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.244981 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.244705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-workload-certs\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.345702 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.345623 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-workload-certs\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.345702 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.345664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-workload-socket\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.345702 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.345684 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-envoy\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.346008 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.345719 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-token\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.346008 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.345758 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-credential-socket\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.346008 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.345788 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-data\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.346008 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.345836 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d40266b-e47c-4c17-8245-e25b227aa17a-istiod-ca-cert\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.346008 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.345871 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqv8r\" (UniqueName: \"kubernetes.io/projected/3d40266b-e47c-4c17-8245-e25b227aa17a-kube-api-access-dqv8r\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.346008 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.345932 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-podinfo\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.346431 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.346398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-workload-certs\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.346632 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.346588 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d40266b-e47c-4c17-8245-e25b227aa17a-istiod-ca-cert\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.346717 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.346673 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-workload-socket\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.346893 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.346860 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-data\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.347123 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.347104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-credential-socket\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.349144 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.349113 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-podinfo\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.349363 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.349343 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-envoy\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.353855 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.353828 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-token\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.354124 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.354103 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqv8r\" (UniqueName: \"kubernetes.io/projected/3d40266b-e47c-4c17-8245-e25b227aa17a-kube-api-access-dqv8r\") pod \"e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.395007 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.394977 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:05.674431 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.674403 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz"] Sep 17 23:22:05.676298 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:22:05.676266 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d40266b_e47c_4c17_8245_e25b227aa17a.slice/crio-9d25886c63573eb35af8566fad5f1ce75157bbad910190e700f119332f609f70 WatchSource:0}: Error finding container 9d25886c63573eb35af8566fad5f1ce75157bbad910190e700f119332f609f70: Status 404 returned error can't find the container with id 9d25886c63573eb35af8566fad5f1ce75157bbad910190e700f119332f609f70 Sep 17 23:22:05.678534 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.678492 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:22:05.678603 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.678576 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:22:05.678651 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:05.678617 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:22:06.229090 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:06.229045 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" event={"ID":"3d40266b-e47c-4c17-8245-e25b227aa17a","Type":"ContainerStarted","Data":"639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93"} Sep 17 23:22:06.229495 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:06.229152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" event={"ID":"3d40266b-e47c-4c17-8245-e25b227aa17a","Type":"ContainerStarted","Data":"9d25886c63573eb35af8566fad5f1ce75157bbad910190e700f119332f609f70"} Sep 17 23:22:06.395694 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:06.395659 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:06.400634 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:06.400604 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:06.430303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:06.430250 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" podStartSLOduration=1.4302346400000001 podStartE2EDuration="1.43023464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:06.269205313 +0000 UTC m=+954.386684762" watchObservedRunningTime="2026-09-17 23:22:06.43023464 +0000 UTC m=+954.547714088" Sep 17 23:22:07.234180 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:07.234148 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:07.235131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:07.235109 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:11.920608 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:11.920564 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k"] Sep 17 23:22:11.925050 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:11.925027 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:11.928261 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:11.928230 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-35df6d-serving-cert\"" Sep 17 23:22:11.928400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:11.928274 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cqmcq\"" Sep 17 23:22:11.928400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:11.928339 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 23:22:11.928536 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:11.928434 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 23:22:11.928536 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:11.928494 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 23:22:11.934775 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:11.934735 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k"] Sep 17 23:22:12.115136 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.115103 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-maas-api-tls\") pod \"maas-api-e2e-explicit-35df6d-66469dd67d-28d9k\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.115308 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.115152 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-openshift-service-ca\") pod \"maas-api-e2e-explicit-35df6d-66469dd67d-28d9k\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.115308 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.115222 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-metrics-certs\") pod \"maas-api-e2e-explicit-35df6d-66469dd67d-28d9k\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.115308 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.115280 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpc6h\" (UniqueName: \"kubernetes.io/projected/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-kube-api-access-fpc6h\") pod \"maas-api-e2e-explicit-35df6d-66469dd67d-28d9k\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.216473 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.216435 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-openshift-service-ca\") pod \"maas-api-e2e-explicit-35df6d-66469dd67d-28d9k\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.216642 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.216487 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-metrics-certs\") pod \"maas-api-e2e-explicit-35df6d-66469dd67d-28d9k\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.216642 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.216556 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpc6h\" (UniqueName: \"kubernetes.io/projected/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-kube-api-access-fpc6h\") pod \"maas-api-e2e-explicit-35df6d-66469dd67d-28d9k\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.216642 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.216623 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-maas-api-tls\") pod \"maas-api-e2e-explicit-35df6d-66469dd67d-28d9k\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.217240 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.217205 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-openshift-service-ca\") pod \"maas-api-e2e-explicit-35df6d-66469dd67d-28d9k\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.218953 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.218933 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-metrics-certs\") pod \"maas-api-e2e-explicit-35df6d-66469dd67d-28d9k\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.219199 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.219179 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-maas-api-tls\") pod \"maas-api-e2e-explicit-35df6d-66469dd67d-28d9k\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.223986 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.223968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpc6h\" (UniqueName: \"kubernetes.io/projected/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-kube-api-access-fpc6h\") pod \"maas-api-e2e-explicit-35df6d-66469dd67d-28d9k\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.239118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.239095 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:12.513489 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.513466 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k"] Sep 17 23:22:12.515236 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:22:12.515199 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25bee6d_a0e7_488e_a1a8_312e1df18b1a.slice/crio-39b032e029d80cc0c5883f2f880516b92ea704aac293ba19f5ebf7a1b49de969 WatchSource:0}: Error finding container 39b032e029d80cc0c5883f2f880516b92ea704aac293ba19f5ebf7a1b49de969: Status 404 returned error can't find the container with id 39b032e029d80cc0c5883f2f880516b92ea704aac293ba19f5ebf7a1b49de969 Sep 17 23:22:12.561183 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.561154 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4"] Sep 17 23:22:12.564798 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.564781 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" Sep 17 23:22:12.567351 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.567329 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-35df6d\"" Sep 17 23:22:12.567458 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.567329 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-35df6d-dockercfg-lgcvn\"" Sep 17 23:22:12.575895 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.575874 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4"] Sep 17 23:22:12.610870 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.610847 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w"] Sep 17 23:22:12.613345 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.613330 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" Sep 17 23:22:12.622782 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.622758 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w"] Sep 17 23:22:12.721253 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.721221 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3cd482-cfac-464e-84d9-8c387c4ce7ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w\" (UID: \"0a3cd482-cfac-464e-84d9-8c387c4ce7ef\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" Sep 17 23:22:12.721412 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.721266 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmpzs\" (UniqueName: \"kubernetes.io/projected/bb9e13d8-2816-475e-b30a-ffbbbfe21638-kube-api-access-rmpzs\") pod \"payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4\" (UID: \"bb9e13d8-2816-475e-b30a-ffbbbfe21638\") " pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" Sep 17 23:22:12.721412 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.721286 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7d6\" (UniqueName: \"kubernetes.io/projected/0a3cd482-cfac-464e-84d9-8c387c4ce7ef-kube-api-access-4j7d6\") pod \"payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w\" (UID: \"0a3cd482-cfac-464e-84d9-8c387c4ce7ef\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" Sep 17 23:22:12.721412 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.721340 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9e13d8-2816-475e-b30a-ffbbbfe21638-plugins-config-volume\") pod \"payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4\" (UID: \"bb9e13d8-2816-475e-b30a-ffbbbfe21638\") " pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" Sep 17 23:22:12.822804 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.822730 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3cd482-cfac-464e-84d9-8c387c4ce7ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w\" (UID: \"0a3cd482-cfac-464e-84d9-8c387c4ce7ef\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" Sep 17 23:22:12.822804 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.822775 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmpzs\" (UniqueName: \"kubernetes.io/projected/bb9e13d8-2816-475e-b30a-ffbbbfe21638-kube-api-access-rmpzs\") pod \"payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4\" (UID: \"bb9e13d8-2816-475e-b30a-ffbbbfe21638\") " pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" Sep 17 23:22:12.822804 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.822797 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7d6\" (UniqueName: \"kubernetes.io/projected/0a3cd482-cfac-464e-84d9-8c387c4ce7ef-kube-api-access-4j7d6\") pod \"payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w\" (UID: \"0a3cd482-cfac-464e-84d9-8c387c4ce7ef\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" Sep 17 23:22:12.823100 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.822836 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9e13d8-2816-475e-b30a-ffbbbfe21638-plugins-config-volume\") pod \"payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4\" (UID: \"bb9e13d8-2816-475e-b30a-ffbbbfe21638\") " pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" Sep 17 23:22:12.823522 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.823502 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9e13d8-2816-475e-b30a-ffbbbfe21638-plugins-config-volume\") pod \"payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4\" (UID: \"bb9e13d8-2816-475e-b30a-ffbbbfe21638\") " pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" Sep 17 23:22:12.823562 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.823541 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3cd482-cfac-464e-84d9-8c387c4ce7ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w\" (UID: \"0a3cd482-cfac-464e-84d9-8c387c4ce7ef\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" Sep 17 23:22:12.830447 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.830427 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmpzs\" (UniqueName: \"kubernetes.io/projected/bb9e13d8-2816-475e-b30a-ffbbbfe21638-kube-api-access-rmpzs\") pod \"payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4\" (UID: \"bb9e13d8-2816-475e-b30a-ffbbbfe21638\") " pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" Sep 17 23:22:12.830554 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.830487 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7d6\" (UniqueName: \"kubernetes.io/projected/0a3cd482-cfac-464e-84d9-8c387c4ce7ef-kube-api-access-4j7d6\") pod \"payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w\" (UID: \"0a3cd482-cfac-464e-84d9-8c387c4ce7ef\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" Sep 17 23:22:12.875559 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.875533 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" Sep 17 23:22:12.923389 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:12.923355 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" Sep 17 23:22:13.150604 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:13.150574 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4"] Sep 17 23:22:13.153891 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:22:13.153860 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9e13d8_2816_475e_b30a_ffbbbfe21638.slice/crio-dc0fbd656c276b76aa23e71f0e1097e1d358b34aeee6a364cd49ddd36f2f3b60 WatchSource:0}: Error finding container dc0fbd656c276b76aa23e71f0e1097e1d358b34aeee6a364cd49ddd36f2f3b60: Status 404 returned error can't find the container with id dc0fbd656c276b76aa23e71f0e1097e1d358b34aeee6a364cd49ddd36f2f3b60 Sep 17 23:22:13.203344 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:13.203279 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w"] Sep 17 23:22:13.206650 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:22:13.206622 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3cd482_cfac_464e_84d9_8c387c4ce7ef.slice/crio-1237801cce6ded435e87f296b7ca939c6121cf06660dd527574a9ecbbea862f8 WatchSource:0}: Error finding container 1237801cce6ded435e87f296b7ca939c6121cf06660dd527574a9ecbbea862f8: Status 404 returned error can't find the container with id 1237801cce6ded435e87f296b7ca939c6121cf06660dd527574a9ecbbea862f8 Sep 17 23:22:13.269434 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:13.269335 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" event={"ID":"bb9e13d8-2816-475e-b30a-ffbbbfe21638","Type":"ContainerStarted","Data":"8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc"} Sep 17 23:22:13.269434 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:13.269396 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" event={"ID":"bb9e13d8-2816-475e-b30a-ffbbbfe21638","Type":"ContainerStarted","Data":"dc0fbd656c276b76aa23e71f0e1097e1d358b34aeee6a364cd49ddd36f2f3b60"} Sep 17 23:22:13.269686 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:13.269445 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" Sep 17 23:22:13.271401 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:13.271344 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" event={"ID":"0a3cd482-cfac-464e-84d9-8c387c4ce7ef","Type":"ContainerStarted","Data":"1237801cce6ded435e87f296b7ca939c6121cf06660dd527574a9ecbbea862f8"} Sep 17 23:22:13.273582 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:13.273558 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" event={"ID":"b25bee6d-a0e7-488e-a1a8-312e1df18b1a","Type":"ContainerStarted","Data":"39b032e029d80cc0c5883f2f880516b92ea704aac293ba19f5ebf7a1b49de969"} Sep 17 23:22:13.287529 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:13.286798 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" podStartSLOduration=1.286784291 podStartE2EDuration="1.286784291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:13.285350088 +0000 UTC m=+961.402829568" watchObservedRunningTime="2026-09-17 23:22:13.286784291 +0000 UTC m=+961.404263740" Sep 17 23:22:14.290517 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:14.290484 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" event={"ID":"0a3cd482-cfac-464e-84d9-8c387c4ce7ef","Type":"ContainerStarted","Data":"dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80"} Sep 17 23:22:14.290879 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:14.290541 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" Sep 17 23:22:14.304807 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:14.304753 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" podStartSLOduration=2.304734251 podStartE2EDuration="2.304734251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:14.302711247 +0000 UTC m=+962.420190723" watchObservedRunningTime="2026-09-17 23:22:14.304734251 +0000 UTC m=+962.422213700" Sep 17 23:22:15.290052 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:15.290015 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" event={"ID":"b25bee6d-a0e7-488e-a1a8-312e1df18b1a","Type":"ContainerStarted","Data":"0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d"} Sep 17 23:22:15.308033 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:15.307987 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" podStartSLOduration=1.876955919 podStartE2EDuration="4.307971768s" totalImagesPullingTime="2.431015849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:15.306746783 +0000 UTC m=+963.424226235" watchObservedRunningTime="2026-09-17 23:22:15.307971768 +0000 UTC m=+963.425451267" Sep 17 23:22:16.294377 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:16.294351 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:21.600256 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:21.600219 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:22:22.305395 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:22.305366 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:22:24.287690 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:24.287661 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" Sep 17 23:22:25.292683 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:25.292593 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" Sep 17 23:22:32.397645 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:32.397610 2559 generic.go:356] "Generic (PLEG): container finished" podID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerID="639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93" exitCode=0 Sep 17 23:22:32.398064 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:32.397682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" event={"ID":"3d40266b-e47c-4c17-8245-e25b227aa17a","Type":"ContainerDied","Data":"639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93"} Sep 17 23:22:32.398116 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:32.398090 2559 scope.go:117] "RemoveContainer" containerID="639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93" Sep 17 23:22:32.399179 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:32.399151 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:22:32.399236 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:32.399218 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:22:32.399275 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:32.399251 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:22:33.405541 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:33.405508 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" event={"ID":"3d40266b-e47c-4c17-8245-e25b227aa17a","Type":"ContainerStarted","Data":"80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484"} Sep 17 23:22:34.396080 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:34.396044 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:34.397359 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:34.397327 2559 patch_prober.go:28] interesting pod/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 17 23:22:34.397505 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:34.397389 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 17 23:22:35.395305 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:35.395269 2559 patch_prober.go:28] interesting pod/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 17 23:22:35.395713 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:35.395342 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 17 23:22:36.395487 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:36.395450 2559 patch_prober.go:28] interesting pod/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 17 23:22:36.395853 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:36.395518 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 17 23:22:37.234997 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:37.234964 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:37.396200 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:37.396155 2559 patch_prober.go:28] interesting pod/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 17 23:22:37.396676 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:37.396238 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 17 23:22:38.553844 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:38.553809 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:38.555768 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:38.555746 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:22:42.905790 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:22:42.905754 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:23:02.115726 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:02.115692 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:23:15.302524 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.302492 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v"] Sep 17 23:23:15.306265 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.306242 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.308686 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.308662 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-85682bb8-serving-cert\"" Sep 17 23:23:15.318595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.318571 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v"] Sep 17 23:23:15.372106 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.372067 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvj89\" (UniqueName: \"kubernetes.io/projected/0777e6a5-0c29-420e-ac10-73fbf61ba03a-kube-api-access-zvj89\") pod \"maas-api-e2e-mt-85682bb8-7db759458-vfq2v\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.372282 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.372142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0777e6a5-0c29-420e-ac10-73fbf61ba03a-maas-api-tls\") pod \"maas-api-e2e-mt-85682bb8-7db759458-vfq2v\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.372282 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.372172 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0777e6a5-0c29-420e-ac10-73fbf61ba03a-openshift-service-ca\") pod \"maas-api-e2e-mt-85682bb8-7db759458-vfq2v\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.372282 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.372210 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0777e6a5-0c29-420e-ac10-73fbf61ba03a-metrics-certs\") pod \"maas-api-e2e-mt-85682bb8-7db759458-vfq2v\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.473343 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.473307 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvj89\" (UniqueName: \"kubernetes.io/projected/0777e6a5-0c29-420e-ac10-73fbf61ba03a-kube-api-access-zvj89\") pod \"maas-api-e2e-mt-85682bb8-7db759458-vfq2v\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.473525 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.473354 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0777e6a5-0c29-420e-ac10-73fbf61ba03a-maas-api-tls\") pod \"maas-api-e2e-mt-85682bb8-7db759458-vfq2v\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.473525 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.473375 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0777e6a5-0c29-420e-ac10-73fbf61ba03a-openshift-service-ca\") pod \"maas-api-e2e-mt-85682bb8-7db759458-vfq2v\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.473525 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.473496 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0777e6a5-0c29-420e-ac10-73fbf61ba03a-metrics-certs\") pod \"maas-api-e2e-mt-85682bb8-7db759458-vfq2v\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.473970 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.473951 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0777e6a5-0c29-420e-ac10-73fbf61ba03a-openshift-service-ca\") pod \"maas-api-e2e-mt-85682bb8-7db759458-vfq2v\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.475882 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.475858 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0777e6a5-0c29-420e-ac10-73fbf61ba03a-maas-api-tls\") pod \"maas-api-e2e-mt-85682bb8-7db759458-vfq2v\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.475974 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.475892 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0777e6a5-0c29-420e-ac10-73fbf61ba03a-metrics-certs\") pod \"maas-api-e2e-mt-85682bb8-7db759458-vfq2v\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.485630 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.485609 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvj89\" (UniqueName: \"kubernetes.io/projected/0777e6a5-0c29-420e-ac10-73fbf61ba03a-kube-api-access-zvj89\") pod \"maas-api-e2e-mt-85682bb8-7db759458-vfq2v\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.619946 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.619833 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:15.895954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.895888 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v"] Sep 17 23:23:15.897954 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:23:15.897926 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0777e6a5_0c29_420e_ac10_73fbf61ba03a.slice/crio-085c8071dd71814399063fec58c93802931f1ca4a4f36538b027c418c48d2fd9 WatchSource:0}: Error finding container 085c8071dd71814399063fec58c93802931f1ca4a4f36538b027c418c48d2fd9: Status 404 returned error can't find the container with id 085c8071dd71814399063fec58c93802931f1ca4a4f36538b027c418c48d2fd9 Sep 17 23:23:15.951444 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.951416 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg"] Sep 17 23:23:15.954837 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.954821 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" Sep 17 23:23:15.957171 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.957151 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-85682bb8\"" Sep 17 23:23:15.957293 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.957239 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-85682bb8-dockercfg-nj6bb\"" Sep 17 23:23:15.967494 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.967473 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg"] Sep 17 23:23:15.978789 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.978760 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321384c3-7732-4577-af3b-606fcd6c863f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg\" (UID: \"321384c3-7732-4577-af3b-606fcd6c863f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" Sep 17 23:23:15.978935 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:15.978835 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brpcx\" (UniqueName: \"kubernetes.io/projected/321384c3-7732-4577-af3b-606fcd6c863f-kube-api-access-brpcx\") pod \"payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg\" (UID: \"321384c3-7732-4577-af3b-606fcd6c863f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" Sep 17 23:23:16.079937 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.079894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brpcx\" (UniqueName: \"kubernetes.io/projected/321384c3-7732-4577-af3b-606fcd6c863f-kube-api-access-brpcx\") pod \"payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg\" (UID: \"321384c3-7732-4577-af3b-606fcd6c863f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" Sep 17 23:23:16.080092 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.080016 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321384c3-7732-4577-af3b-606fcd6c863f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg\" (UID: \"321384c3-7732-4577-af3b-606fcd6c863f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" Sep 17 23:23:16.080677 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.080659 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321384c3-7732-4577-af3b-606fcd6c863f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg\" (UID: \"321384c3-7732-4577-af3b-606fcd6c863f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" Sep 17 23:23:16.087963 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.087942 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brpcx\" (UniqueName: \"kubernetes.io/projected/321384c3-7732-4577-af3b-606fcd6c863f-kube-api-access-brpcx\") pod \"payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg\" (UID: \"321384c3-7732-4577-af3b-606fcd6c863f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" Sep 17 23:23:16.265138 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.265107 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" Sep 17 23:23:16.538808 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.538778 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg"] Sep 17 23:23:16.539301 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:23:16.539277 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321384c3_7732_4577_af3b_606fcd6c863f.slice/crio-7f68b72b5e01cae43f7dd460b005ca4e09d2e45e9be1319a03b5b7077a6dbf3d WatchSource:0}: Error finding container 7f68b72b5e01cae43f7dd460b005ca4e09d2e45e9be1319a03b5b7077a6dbf3d: Status 404 returned error can't find the container with id 7f68b72b5e01cae43f7dd460b005ca4e09d2e45e9be1319a03b5b7077a6dbf3d Sep 17 23:23:16.637923 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.637872 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" event={"ID":"321384c3-7732-4577-af3b-606fcd6c863f","Type":"ContainerStarted","Data":"bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2"} Sep 17 23:23:16.638082 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.637956 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" Sep 17 23:23:16.638082 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.637978 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" event={"ID":"321384c3-7732-4577-af3b-606fcd6c863f","Type":"ContainerStarted","Data":"7f68b72b5e01cae43f7dd460b005ca4e09d2e45e9be1319a03b5b7077a6dbf3d"} Sep 17 23:23:16.639454 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.639431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" event={"ID":"0777e6a5-0c29-420e-ac10-73fbf61ba03a","Type":"ContainerStarted","Data":"47143a41d1a131587555a9a4005bac66a9e4e694d780e24daeb2ac93f4ebe109"} Sep 17 23:23:16.639537 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.639478 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" event={"ID":"0777e6a5-0c29-420e-ac10-73fbf61ba03a","Type":"ContainerStarted","Data":"085c8071dd71814399063fec58c93802931f1ca4a4f36538b027c418c48d2fd9"} Sep 17 23:23:16.639537 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.639501 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:16.656949 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.656882 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" podStartSLOduration=1.656866812 podStartE2EDuration="1.656866812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:16.656556218 +0000 UTC m=+1024.774035680" watchObservedRunningTime="2026-09-17 23:23:16.656866812 +0000 UTC m=+1024.774346261" Sep 17 23:23:16.676189 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:16.676140 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" podStartSLOduration=1.136008536 podStartE2EDuration="1.676123195s" totalImagesPullingTime="540.114659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:16.674675345 +0000 UTC m=+1024.792154791" watchObservedRunningTime="2026-09-17 23:23:16.676123195 +0000 UTC m=+1024.793602644" Sep 17 23:23:22.652821 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:22.652793 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:27.649550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:27.649520 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" Sep 17 23:23:33.395269 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:33.395231 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:23:58.741205 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:58.740997 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v"] Sep 17 23:23:58.742005 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:58.741969 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" podUID="0777e6a5-0c29-420e-ac10-73fbf61ba03a" containerName="maas-api" containerID="cri-o://47143a41d1a131587555a9a4005bac66a9e4e694d780e24daeb2ac93f4ebe109" gracePeriod=30 Sep 17 23:23:58.788945 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:58.788879 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg"] Sep 17 23:23:58.789213 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:58.789156 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" podUID="321384c3-7732-4577-af3b-606fcd6c863f" containerName="payload-pre-processing" containerID="cri-o://bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2" gracePeriod=30 Sep 17 23:23:58.874321 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:58.874292 2559 generic.go:356] "Generic (PLEG): container finished" podID="0777e6a5-0c29-420e-ac10-73fbf61ba03a" containerID="47143a41d1a131587555a9a4005bac66a9e4e694d780e24daeb2ac93f4ebe109" exitCode=0 Sep 17 23:23:58.874461 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:58.874360 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" event={"ID":"0777e6a5-0c29-420e-ac10-73fbf61ba03a","Type":"ContainerDied","Data":"47143a41d1a131587555a9a4005bac66a9e4e694d780e24daeb2ac93f4ebe109"} Sep 17 23:23:59.046184 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.045854 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:59.058914 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.058873 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0777e6a5-0c29-420e-ac10-73fbf61ba03a-maas-api-tls\") pod \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " Sep 17 23:23:59.059059 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.058927 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvj89\" (UniqueName: \"kubernetes.io/projected/0777e6a5-0c29-420e-ac10-73fbf61ba03a-kube-api-access-zvj89\") pod \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " Sep 17 23:23:59.059059 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.058943 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0777e6a5-0c29-420e-ac10-73fbf61ba03a-metrics-certs\") pod \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " Sep 17 23:23:59.059059 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.058960 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0777e6a5-0c29-420e-ac10-73fbf61ba03a-openshift-service-ca\") pod \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\" (UID: \"0777e6a5-0c29-420e-ac10-73fbf61ba03a\") " Sep 17 23:23:59.059434 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.059406 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0777e6a5-0c29-420e-ac10-73fbf61ba03a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0777e6a5-0c29-420e-ac10-73fbf61ba03a" (UID: "0777e6a5-0c29-420e-ac10-73fbf61ba03a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:23:59.061176 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.061122 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0777e6a5-0c29-420e-ac10-73fbf61ba03a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0777e6a5-0c29-420e-ac10-73fbf61ba03a" (UID: "0777e6a5-0c29-420e-ac10-73fbf61ba03a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:23:59.061341 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.061317 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0777e6a5-0c29-420e-ac10-73fbf61ba03a-kube-api-access-zvj89" (OuterVolumeSpecName: "kube-api-access-zvj89") pod "0777e6a5-0c29-420e-ac10-73fbf61ba03a" (UID: "0777e6a5-0c29-420e-ac10-73fbf61ba03a"). InnerVolumeSpecName "kube-api-access-zvj89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:23:59.061341 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.061330 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0777e6a5-0c29-420e-ac10-73fbf61ba03a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0777e6a5-0c29-420e-ac10-73fbf61ba03a" (UID: "0777e6a5-0c29-420e-ac10-73fbf61ba03a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:23:59.085272 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.085247 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" Sep 17 23:23:59.160018 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.159992 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321384c3-7732-4577-af3b-606fcd6c863f-plugins-config-volume\") pod \"321384c3-7732-4577-af3b-606fcd6c863f\" (UID: \"321384c3-7732-4577-af3b-606fcd6c863f\") " Sep 17 23:23:59.160167 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.160080 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brpcx\" (UniqueName: \"kubernetes.io/projected/321384c3-7732-4577-af3b-606fcd6c863f-kube-api-access-brpcx\") pod \"321384c3-7732-4577-af3b-606fcd6c863f\" (UID: \"321384c3-7732-4577-af3b-606fcd6c863f\") " Sep 17 23:23:59.160262 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.160251 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0777e6a5-0c29-420e-ac10-73fbf61ba03a-openshift-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:23:59.160326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.160263 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0777e6a5-0c29-420e-ac10-73fbf61ba03a-maas-api-tls\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:23:59.160326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.160272 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvj89\" (UniqueName: \"kubernetes.io/projected/0777e6a5-0c29-420e-ac10-73fbf61ba03a-kube-api-access-zvj89\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:23:59.160326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.160279 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0777e6a5-0c29-420e-ac10-73fbf61ba03a-metrics-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:23:59.160326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.160317 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321384c3-7732-4577-af3b-606fcd6c863f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "321384c3-7732-4577-af3b-606fcd6c863f" (UID: "321384c3-7732-4577-af3b-606fcd6c863f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:23:59.162183 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.162161 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321384c3-7732-4577-af3b-606fcd6c863f-kube-api-access-brpcx" (OuterVolumeSpecName: "kube-api-access-brpcx") pod "321384c3-7732-4577-af3b-606fcd6c863f" (UID: "321384c3-7732-4577-af3b-606fcd6c863f"). InnerVolumeSpecName "kube-api-access-brpcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:23:59.261307 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.261274 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321384c3-7732-4577-af3b-606fcd6c863f-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:23:59.261307 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.261302 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brpcx\" (UniqueName: \"kubernetes.io/projected/321384c3-7732-4577-af3b-606fcd6c863f-kube-api-access-brpcx\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:23:59.881997 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.881967 2559 generic.go:356] "Generic (PLEG): container finished" podID="321384c3-7732-4577-af3b-606fcd6c863f" containerID="bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2" exitCode=0 Sep 17 23:23:59.882432 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.882010 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" Sep 17 23:23:59.882432 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.882055 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" event={"ID":"321384c3-7732-4577-af3b-606fcd6c863f","Type":"ContainerDied","Data":"bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2"} Sep 17 23:23:59.882432 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.882120 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg" event={"ID":"321384c3-7732-4577-af3b-606fcd6c863f","Type":"ContainerDied","Data":"7f68b72b5e01cae43f7dd460b005ca4e09d2e45e9be1319a03b5b7077a6dbf3d"} Sep 17 23:23:59.882432 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.882152 2559 scope.go:117] "RemoveContainer" containerID="bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2" Sep 17 23:23:59.883523 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.883504 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" event={"ID":"0777e6a5-0c29-420e-ac10-73fbf61ba03a","Type":"ContainerDied","Data":"085c8071dd71814399063fec58c93802931f1ca4a4f36538b027c418c48d2fd9"} Sep 17 23:23:59.883602 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.883524 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v" Sep 17 23:23:59.891308 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.891291 2559 scope.go:117] "RemoveContainer" containerID="bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2" Sep 17 23:23:59.891622 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:23:59.891597 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2\": container with ID starting with bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2 not found: ID does not exist" containerID="bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2" Sep 17 23:23:59.891719 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.891631 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2"} err="failed to get container status \"bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2\": rpc error: code = NotFound desc = could not find container \"bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2\": container with ID starting with bf5b6d64036cc8c6e3dd2f6adbb3e70ccc8424c42610cf7419b72967133567b2 not found: ID does not exist" Sep 17 23:23:59.891719 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.891656 2559 scope.go:117] "RemoveContainer" containerID="47143a41d1a131587555a9a4005bac66a9e4e694d780e24daeb2ac93f4ebe109" Sep 17 23:23:59.911744 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.911717 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v"] Sep 17 23:23:59.917219 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.917197 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85682bb8-7db759458-vfq2v"] Sep 17 23:23:59.927171 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.927153 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg"] Sep 17 23:23:59.931122 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:23:59.931104 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85682bb8-77d8b6d74b-52nbg"] Sep 17 23:24:00.288762 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:00.288728 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0777e6a5-0c29-420e-ac10-73fbf61ba03a" path="/var/lib/kubelet/pods/0777e6a5-0c29-420e-ac10-73fbf61ba03a/volumes" Sep 17 23:24:00.289213 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:00.289192 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="321384c3-7732-4577-af3b-606fcd6c863f" path="/var/lib/kubelet/pods/321384c3-7732-4577-af3b-606fcd6c863f/volumes" Sep 17 23:24:08.002094 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:08.002061 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:24:19.038420 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.038377 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd"] Sep 17 23:24:19.039139 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.039116 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="321384c3-7732-4577-af3b-606fcd6c863f" containerName="payload-pre-processing" Sep 17 23:24:19.039216 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.039143 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="321384c3-7732-4577-af3b-606fcd6c863f" containerName="payload-pre-processing" Sep 17 23:24:19.039216 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.039177 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0777e6a5-0c29-420e-ac10-73fbf61ba03a" containerName="maas-api" Sep 17 23:24:19.039216 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.039186 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0777e6a5-0c29-420e-ac10-73fbf61ba03a" containerName="maas-api" Sep 17 23:24:19.039326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.039310 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="321384c3-7732-4577-af3b-606fcd6c863f" containerName="payload-pre-processing" Sep 17 23:24:19.039370 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.039330 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0777e6a5-0c29-420e-ac10-73fbf61ba03a" containerName="maas-api" Sep 17 23:24:19.043502 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.043479 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.046022 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.046003 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b52922-openshift-default-dockercfg-294qx\"" Sep 17 23:24:19.054172 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.054145 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd"] Sep 17 23:24:19.131565 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.131535 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-istio-envoy\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.131740 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.131580 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59837b90-41e0-4b62-a77f-5b100840f529-istio-token\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.131740 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.131665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59837b90-41e0-4b62-a77f-5b100840f529-istio-podinfo\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.131740 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.131706 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59837b90-41e0-4b62-a77f-5b100840f529-istiod-ca-cert\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.131740 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.131736 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-workload-certs\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.131976 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.131793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-workload-socket\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.131976 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.131817 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2xqs\" (UniqueName: \"kubernetes.io/projected/59837b90-41e0-4b62-a77f-5b100840f529-kube-api-access-g2xqs\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.131976 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.131857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-credential-socket\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.131976 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.131888 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-istio-data\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233039 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59837b90-41e0-4b62-a77f-5b100840f529-istio-podinfo\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233215 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59837b90-41e0-4b62-a77f-5b100840f529-istiod-ca-cert\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233215 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233069 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-workload-certs\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233215 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233106 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-workload-socket\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233215 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2xqs\" (UniqueName: \"kubernetes.io/projected/59837b90-41e0-4b62-a77f-5b100840f529-kube-api-access-g2xqs\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233215 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233161 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-credential-socket\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233496 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233294 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-istio-data\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233496 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233355 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-istio-envoy\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233496 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233406 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59837b90-41e0-4b62-a77f-5b100840f529-istio-token\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233786 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233752 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-credential-socket\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233844 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233774 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59837b90-41e0-4b62-a77f-5b100840f529-istiod-ca-cert\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233844 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233812 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-workload-certs\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.233981 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.233872 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-istio-data\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.234074 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.234059 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-workload-socket\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.235548 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.235524 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59837b90-41e0-4b62-a77f-5b100840f529-istio-podinfo\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.235781 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.235762 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-istio-envoy\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.241617 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.241576 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59837b90-41e0-4b62-a77f-5b100840f529-istio-token\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.242058 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.242035 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2xqs\" (UniqueName: \"kubernetes.io/projected/59837b90-41e0-4b62-a77f-5b100840f529-kube-api-access-g2xqs\") pod \"e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.357296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.357222 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:19.630799 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.630686 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd"] Sep 17 23:24:19.633713 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:24:19.633685 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59837b90_41e0_4b62_a77f_5b100840f529.slice/crio-76c7470d2b74be5bd52c4ecb73eea9d3d663a5711e725dfa5b6538369bf66133 WatchSource:0}: Error finding container 76c7470d2b74be5bd52c4ecb73eea9d3d663a5711e725dfa5b6538369bf66133: Status 404 returned error can't find the container with id 76c7470d2b74be5bd52c4ecb73eea9d3d663a5711e725dfa5b6538369bf66133 Sep 17 23:24:19.636487 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.636456 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:24:19.636558 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.636520 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:24:19.636558 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.636545 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:24:19.993443 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.993403 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" event={"ID":"59837b90-41e0-4b62-a77f-5b100840f529","Type":"ContainerStarted","Data":"0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247"} Sep 17 23:24:19.993615 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:19.993461 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" event={"ID":"59837b90-41e0-4b62-a77f-5b100840f529","Type":"ContainerStarted","Data":"76c7470d2b74be5bd52c4ecb73eea9d3d663a5711e725dfa5b6538369bf66133"} Sep 17 23:24:20.020139 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:20.020089 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" podStartSLOduration=1.020075203 podStartE2EDuration="1.020075203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:20.018989966 +0000 UTC m=+1088.136469433" watchObservedRunningTime="2026-09-17 23:24:20.020075203 +0000 UTC m=+1088.137554652" Sep 17 23:24:20.358262 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:20.358181 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:20.363323 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:20.363294 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:21.000453 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:21.000424 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:21.001236 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:21.001213 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:25.986586 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:25.986552 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k"] Sep 17 23:24:25.987067 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:25.986825 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" podUID="b25bee6d-a0e7-488e-a1a8-312e1df18b1a" containerName="maas-api" containerID="cri-o://0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d" gracePeriod=30 Sep 17 23:24:26.049895 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.049864 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4"] Sep 17 23:24:26.050135 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.050107 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" podUID="bb9e13d8-2816-475e-b30a-ffbbbfe21638" containerName="payload-processing" containerID="cri-o://8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc" gracePeriod=30 Sep 17 23:24:26.057802 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.057762 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w"] Sep 17 23:24:26.058029 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.057999 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" podUID="0a3cd482-cfac-464e-84d9-8c387c4ce7ef" containerName="payload-pre-processing" containerID="cri-o://dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80" gracePeriod=30 Sep 17 23:24:26.378498 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.378476 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" Sep 17 23:24:26.382347 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.382330 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:24:26.386087 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.386070 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" Sep 17 23:24:26.495032 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.495002 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-openshift-service-ca\") pod \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " Sep 17 23:24:26.495032 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.495032 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpc6h\" (UniqueName: \"kubernetes.io/projected/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-kube-api-access-fpc6h\") pod \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " Sep 17 23:24:26.495270 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.495063 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmpzs\" (UniqueName: \"kubernetes.io/projected/bb9e13d8-2816-475e-b30a-ffbbbfe21638-kube-api-access-rmpzs\") pod \"bb9e13d8-2816-475e-b30a-ffbbbfe21638\" (UID: \"bb9e13d8-2816-475e-b30a-ffbbbfe21638\") " Sep 17 23:24:26.495270 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.495091 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3cd482-cfac-464e-84d9-8c387c4ce7ef-plugins-config-volume\") pod \"0a3cd482-cfac-464e-84d9-8c387c4ce7ef\" (UID: \"0a3cd482-cfac-464e-84d9-8c387c4ce7ef\") " Sep 17 23:24:26.495270 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.495197 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4j7d6\" (UniqueName: \"kubernetes.io/projected/0a3cd482-cfac-464e-84d9-8c387c4ce7ef-kube-api-access-4j7d6\") pod \"0a3cd482-cfac-464e-84d9-8c387c4ce7ef\" (UID: \"0a3cd482-cfac-464e-84d9-8c387c4ce7ef\") " Sep 17 23:24:26.495430 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.495326 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-maas-api-tls\") pod \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " Sep 17 23:24:26.495430 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.495353 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-metrics-certs\") pod \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\" (UID: \"b25bee6d-a0e7-488e-a1a8-312e1df18b1a\") " Sep 17 23:24:26.495430 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.495358 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b25bee6d-a0e7-488e-a1a8-312e1df18b1a" (UID: "b25bee6d-a0e7-488e-a1a8-312e1df18b1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:24:26.495430 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.495410 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9e13d8-2816-475e-b30a-ffbbbfe21638-plugins-config-volume\") pod \"bb9e13d8-2816-475e-b30a-ffbbbfe21638\" (UID: \"bb9e13d8-2816-475e-b30a-ffbbbfe21638\") " Sep 17 23:24:26.495625 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.495444 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3cd482-cfac-464e-84d9-8c387c4ce7ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a3cd482-cfac-464e-84d9-8c387c4ce7ef" (UID: "0a3cd482-cfac-464e-84d9-8c387c4ce7ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:24:26.495752 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.495735 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-openshift-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:26.495808 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.495753 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3cd482-cfac-464e-84d9-8c387c4ce7ef-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:26.496025 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.496000 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9e13d8-2816-475e-b30a-ffbbbfe21638-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb9e13d8-2816-475e-b30a-ffbbbfe21638" (UID: "bb9e13d8-2816-475e-b30a-ffbbbfe21638"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:24:26.497346 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.497319 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9e13d8-2816-475e-b30a-ffbbbfe21638-kube-api-access-rmpzs" (OuterVolumeSpecName: "kube-api-access-rmpzs") pod "bb9e13d8-2816-475e-b30a-ffbbbfe21638" (UID: "bb9e13d8-2816-475e-b30a-ffbbbfe21638"). InnerVolumeSpecName "kube-api-access-rmpzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:24:26.497492 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.497472 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b25bee6d-a0e7-488e-a1a8-312e1df18b1a" (UID: "b25bee6d-a0e7-488e-a1a8-312e1df18b1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:24:26.497621 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.497601 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-kube-api-access-fpc6h" (OuterVolumeSpecName: "kube-api-access-fpc6h") pod "b25bee6d-a0e7-488e-a1a8-312e1df18b1a" (UID: "b25bee6d-a0e7-488e-a1a8-312e1df18b1a"). InnerVolumeSpecName "kube-api-access-fpc6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:24:26.497715 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.497695 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3cd482-cfac-464e-84d9-8c387c4ce7ef-kube-api-access-4j7d6" (OuterVolumeSpecName: "kube-api-access-4j7d6") pod "0a3cd482-cfac-464e-84d9-8c387c4ce7ef" (UID: "0a3cd482-cfac-464e-84d9-8c387c4ce7ef"). InnerVolumeSpecName "kube-api-access-4j7d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:24:26.497763 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.497738 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b25bee6d-a0e7-488e-a1a8-312e1df18b1a" (UID: "b25bee6d-a0e7-488e-a1a8-312e1df18b1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:24:26.596536 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.596477 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-maas-api-tls\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:26.596536 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.596498 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-metrics-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:26.596536 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.596508 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9e13d8-2816-475e-b30a-ffbbbfe21638-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:26.596536 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.596517 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpc6h\" (UniqueName: \"kubernetes.io/projected/b25bee6d-a0e7-488e-a1a8-312e1df18b1a-kube-api-access-fpc6h\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:26.596536 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.596525 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmpzs\" (UniqueName: \"kubernetes.io/projected/bb9e13d8-2816-475e-b30a-ffbbbfe21638-kube-api-access-rmpzs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:26.596536 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:26.596535 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j7d6\" (UniqueName: \"kubernetes.io/projected/0a3cd482-cfac-464e-84d9-8c387c4ce7ef-kube-api-access-4j7d6\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:27.033918 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.033868 2559 generic.go:356] "Generic (PLEG): container finished" podID="bb9e13d8-2816-475e-b30a-ffbbbfe21638" containerID="8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc" exitCode=0 Sep 17 23:24:27.034401 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.033939 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" Sep 17 23:24:27.034401 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.033969 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" event={"ID":"bb9e13d8-2816-475e-b30a-ffbbbfe21638","Type":"ContainerDied","Data":"8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc"} Sep 17 23:24:27.034401 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.034038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4" event={"ID":"bb9e13d8-2816-475e-b30a-ffbbbfe21638","Type":"ContainerDied","Data":"dc0fbd656c276b76aa23e71f0e1097e1d358b34aeee6a364cd49ddd36f2f3b60"} Sep 17 23:24:27.034401 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.034061 2559 scope.go:117] "RemoveContainer" containerID="8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc" Sep 17 23:24:27.036012 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.035984 2559 generic.go:356] "Generic (PLEG): container finished" podID="0a3cd482-cfac-464e-84d9-8c387c4ce7ef" containerID="dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80" exitCode=0 Sep 17 23:24:27.036161 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.036077 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" event={"ID":"0a3cd482-cfac-464e-84d9-8c387c4ce7ef","Type":"ContainerDied","Data":"dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80"} Sep 17 23:24:27.036161 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.036117 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" event={"ID":"0a3cd482-cfac-464e-84d9-8c387c4ce7ef","Type":"ContainerDied","Data":"1237801cce6ded435e87f296b7ca939c6121cf06660dd527574a9ecbbea862f8"} Sep 17 23:24:27.036271 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.036189 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w" Sep 17 23:24:27.037847 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.037825 2559 generic.go:356] "Generic (PLEG): container finished" podID="b25bee6d-a0e7-488e-a1a8-312e1df18b1a" containerID="0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d" exitCode=0 Sep 17 23:24:27.037976 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.037862 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" Sep 17 23:24:27.037976 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.037859 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" event={"ID":"b25bee6d-a0e7-488e-a1a8-312e1df18b1a","Type":"ContainerDied","Data":"0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d"} Sep 17 23:24:27.037976 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.037943 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k" event={"ID":"b25bee6d-a0e7-488e-a1a8-312e1df18b1a","Type":"ContainerDied","Data":"39b032e029d80cc0c5883f2f880516b92ea704aac293ba19f5ebf7a1b49de969"} Sep 17 23:24:27.038159 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.038091 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh"] Sep 17 23:24:27.038705 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.038687 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb9e13d8-2816-475e-b30a-ffbbbfe21638" containerName="payload-processing" Sep 17 23:24:27.038820 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.038707 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb9e13d8-2816-475e-b30a-ffbbbfe21638" containerName="payload-processing" Sep 17 23:24:27.038820 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.038734 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3cd482-cfac-464e-84d9-8c387c4ce7ef" containerName="payload-pre-processing" Sep 17 23:24:27.038820 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.038743 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3cd482-cfac-464e-84d9-8c387c4ce7ef" containerName="payload-pre-processing" Sep 17 23:24:27.038820 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.038756 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25bee6d-a0e7-488e-a1a8-312e1df18b1a" containerName="maas-api" Sep 17 23:24:27.038820 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.038767 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25bee6d-a0e7-488e-a1a8-312e1df18b1a" containerName="maas-api" Sep 17 23:24:27.039054 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.038868 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25bee6d-a0e7-488e-a1a8-312e1df18b1a" containerName="maas-api" Sep 17 23:24:27.039054 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.038880 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a3cd482-cfac-464e-84d9-8c387c4ce7ef" containerName="payload-pre-processing" Sep 17 23:24:27.039054 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.038895 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb9e13d8-2816-475e-b30a-ffbbbfe21638" containerName="payload-processing" Sep 17 23:24:27.055289 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.049807 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv"] Sep 17 23:24:27.055289 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.054024 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" Sep 17 23:24:27.058056 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.057223 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b52922\"" Sep 17 23:24:27.058056 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.058039 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh"] Sep 17 23:24:27.058394 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.058141 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" Sep 17 23:24:27.058394 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.058145 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b52922-dockercfg-pdf8p\"" Sep 17 23:24:27.064373 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.063386 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv"] Sep 17 23:24:27.071465 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.069042 2559 scope.go:117] "RemoveContainer" containerID="8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc" Sep 17 23:24:27.071465 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:24:27.071144 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc\": container with ID starting with 8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc not found: ID does not exist" containerID="8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc" Sep 17 23:24:27.071465 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.071191 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc"} err="failed to get container status \"8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc\": rpc error: code = NotFound desc = could not find container \"8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc\": container with ID starting with 8e9a125d9fc8272c3db3ca376357e5b8fec33e053447f9b916b3903e9a46d7dc not found: ID does not exist" Sep 17 23:24:27.071465 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.071217 2559 scope.go:117] "RemoveContainer" containerID="dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80" Sep 17 23:24:27.085051 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.085027 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k"] Sep 17 23:24:27.085823 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.085805 2559 scope.go:117] "RemoveContainer" containerID="dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80" Sep 17 23:24:27.086218 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:24:27.086192 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80\": container with ID starting with dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80 not found: ID does not exist" containerID="dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80" Sep 17 23:24:27.086307 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.086227 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80"} err="failed to get container status \"dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80\": rpc error: code = NotFound desc = could not find container \"dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80\": container with ID starting with dcccb80d8d4919b2939f133cd24789e45a4cd885b22b917cc7276d3d61cb3d80 not found: ID does not exist" Sep 17 23:24:27.086307 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.086250 2559 scope.go:117] "RemoveContainer" containerID="0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d" Sep 17 23:24:27.087173 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.087151 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-35df6d-66469dd67d-28d9k"] Sep 17 23:24:27.096304 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.096287 2559 scope.go:117] "RemoveContainer" containerID="0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d" Sep 17 23:24:27.096564 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:24:27.096545 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d\": container with ID starting with 0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d not found: ID does not exist" containerID="0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d" Sep 17 23:24:27.096642 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.096574 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d"} err="failed to get container status \"0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d\": rpc error: code = NotFound desc = could not find container \"0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d\": container with ID starting with 0d9af7f805be0089877f909ccf9103e8d86bd9be57a53bb995f5d2eb4e6c1d2d not found: ID does not exist" Sep 17 23:24:27.111935 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.111888 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4"] Sep 17 23:24:27.115146 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.115120 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-35df6d-66f95dfccc-gttn4"] Sep 17 23:24:27.139786 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.139760 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w"] Sep 17 23:24:27.145587 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.145443 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-35df6d-c96747fc4-bh96w"] Sep 17 23:24:27.201946 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.201889 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d8d906-d030-4731-94ac-3522ad0b41bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv\" (UID: \"45d8d906-d030-4731-94ac-3522ad0b41bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" Sep 17 23:24:27.202107 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.201997 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsnzb\" (UniqueName: \"kubernetes.io/projected/d67396b8-6a45-456f-b541-45defaca6bf6-kube-api-access-zsnzb\") pod \"payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh\" (UID: \"d67396b8-6a45-456f-b541-45defaca6bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" Sep 17 23:24:27.202107 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.202018 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c828s\" (UniqueName: \"kubernetes.io/projected/45d8d906-d030-4731-94ac-3522ad0b41bb-kube-api-access-c828s\") pod \"payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv\" (UID: \"45d8d906-d030-4731-94ac-3522ad0b41bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" Sep 17 23:24:27.202107 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.202074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67396b8-6a45-456f-b541-45defaca6bf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh\" (UID: \"d67396b8-6a45-456f-b541-45defaca6bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" Sep 17 23:24:27.302567 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.302478 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d8d906-d030-4731-94ac-3522ad0b41bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv\" (UID: \"45d8d906-d030-4731-94ac-3522ad0b41bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" Sep 17 23:24:27.302567 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.302535 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsnzb\" (UniqueName: \"kubernetes.io/projected/d67396b8-6a45-456f-b541-45defaca6bf6-kube-api-access-zsnzb\") pod \"payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh\" (UID: \"d67396b8-6a45-456f-b541-45defaca6bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" Sep 17 23:24:27.302567 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.302552 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c828s\" (UniqueName: \"kubernetes.io/projected/45d8d906-d030-4731-94ac-3522ad0b41bb-kube-api-access-c828s\") pod \"payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv\" (UID: \"45d8d906-d030-4731-94ac-3522ad0b41bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" Sep 17 23:24:27.302797 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.302621 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67396b8-6a45-456f-b541-45defaca6bf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh\" (UID: \"d67396b8-6a45-456f-b541-45defaca6bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" Sep 17 23:24:27.303432 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.303412 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d8d906-d030-4731-94ac-3522ad0b41bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv\" (UID: \"45d8d906-d030-4731-94ac-3522ad0b41bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" Sep 17 23:24:27.303510 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.303489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67396b8-6a45-456f-b541-45defaca6bf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh\" (UID: \"d67396b8-6a45-456f-b541-45defaca6bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" Sep 17 23:24:27.323778 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.323693 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c828s\" (UniqueName: \"kubernetes.io/projected/45d8d906-d030-4731-94ac-3522ad0b41bb-kube-api-access-c828s\") pod \"payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv\" (UID: \"45d8d906-d030-4731-94ac-3522ad0b41bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" Sep 17 23:24:27.326363 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.326329 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsnzb\" (UniqueName: \"kubernetes.io/projected/d67396b8-6a45-456f-b541-45defaca6bf6-kube-api-access-zsnzb\") pod \"payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh\" (UID: \"d67396b8-6a45-456f-b541-45defaca6bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" Sep 17 23:24:27.378308 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.378276 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" Sep 17 23:24:27.387171 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.387141 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" Sep 17 23:24:27.662563 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.662535 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh"] Sep 17 23:24:27.664215 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:24:27.664183 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67396b8_6a45_456f_b541_45defaca6bf6.slice/crio-7fac441904497083605f2ed68b9eca6c00d86635b59a1bb1ef80b8f2625a9733 WatchSource:0}: Error finding container 7fac441904497083605f2ed68b9eca6c00d86635b59a1bb1ef80b8f2625a9733: Status 404 returned error can't find the container with id 7fac441904497083605f2ed68b9eca6c00d86635b59a1bb1ef80b8f2625a9733 Sep 17 23:24:27.674225 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:27.674206 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv"] Sep 17 23:24:27.676482 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:24:27.676456 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d8d906_d030_4731_94ac_3522ad0b41bb.slice/crio-495e453ab2d7ceb985bbc42b95ecc0bdbe20b044c329711fffbaf95e2b9efa2f WatchSource:0}: Error finding container 495e453ab2d7ceb985bbc42b95ecc0bdbe20b044c329711fffbaf95e2b9efa2f: Status 404 returned error can't find the container with id 495e453ab2d7ceb985bbc42b95ecc0bdbe20b044c329711fffbaf95e2b9efa2f Sep 17 23:24:28.047740 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:28.047705 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" event={"ID":"d67396b8-6a45-456f-b541-45defaca6bf6","Type":"ContainerStarted","Data":"ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce"} Sep 17 23:24:28.048235 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:28.047760 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" event={"ID":"d67396b8-6a45-456f-b541-45defaca6bf6","Type":"ContainerStarted","Data":"7fac441904497083605f2ed68b9eca6c00d86635b59a1bb1ef80b8f2625a9733"} Sep 17 23:24:28.048235 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:28.047822 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" Sep 17 23:24:28.049731 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:28.049701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" event={"ID":"45d8d906-d030-4731-94ac-3522ad0b41bb","Type":"ContainerStarted","Data":"0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c"} Sep 17 23:24:28.049850 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:28.049754 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" Sep 17 23:24:28.049850 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:28.049776 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" event={"ID":"45d8d906-d030-4731-94ac-3522ad0b41bb","Type":"ContainerStarted","Data":"495e453ab2d7ceb985bbc42b95ecc0bdbe20b044c329711fffbaf95e2b9efa2f"} Sep 17 23:24:28.064817 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:28.064773 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" podStartSLOduration=1.064760258 podStartE2EDuration="1.064760258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:28.063216666 +0000 UTC m=+1096.180696134" watchObservedRunningTime="2026-09-17 23:24:28.064760258 +0000 UTC m=+1096.182239707" Sep 17 23:24:28.289156 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:28.289125 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a3cd482-cfac-464e-84d9-8c387c4ce7ef" path="/var/lib/kubelet/pods/0a3cd482-cfac-464e-84d9-8c387c4ce7ef/volumes" Sep 17 23:24:28.289612 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:28.289593 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b25bee6d-a0e7-488e-a1a8-312e1df18b1a" path="/var/lib/kubelet/pods/b25bee6d-a0e7-488e-a1a8-312e1df18b1a/volumes" Sep 17 23:24:28.289983 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:28.289965 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb9e13d8-2816-475e-b30a-ffbbbfe21638" path="/var/lib/kubelet/pods/bb9e13d8-2816-475e-b30a-ffbbbfe21638/volumes" Sep 17 23:24:38.086893 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:38.086840 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" podStartSLOduration=11.086822447 podStartE2EDuration="11.086822447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:28.084369329 +0000 UTC m=+1096.201848780" watchObservedRunningTime="2026-09-17 23:24:38.086822447 +0000 UTC m=+1106.204301912" Sep 17 23:24:38.088751 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:38.088732 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz"] Sep 17 23:24:38.088960 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:38.088937 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerName="istio-proxy" containerID="cri-o://80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484" gracePeriod=30 Sep 17 23:24:38.556760 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:38.556719 2559 patch_prober.go:28] interesting pod/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 23:24:38.556968 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:38.556793 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 23:24:38.824944 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:38.824842 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d"] Sep 17 23:24:38.828756 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:38.828730 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:38.831229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:38.831208 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-be3eb4-openshift-default-dockercfg-z2pvz\"" Sep 17 23:24:38.842635 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:38.842554 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d"] Sep 17 23:24:39.012709 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.012671 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-credential-socket\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.012870 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.012713 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6009324f-0107-4cd3-9890-9fa7521de183-istiod-ca-cert\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.012870 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.012777 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-workload-certs\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.012870 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.012839 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-istio-envoy\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.013012 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.012885 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-istio-data\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.013094 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.012945 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-workload-socket\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.013132 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.013106 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6009324f-0107-4cd3-9890-9fa7521de183-istio-token\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.013165 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.013153 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6009324f-0107-4cd3-9890-9fa7521de183-istio-podinfo\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.013226 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.013213 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhbx\" (UniqueName: \"kubernetes.io/projected/6009324f-0107-4cd3-9890-9fa7521de183-kube-api-access-zbhbx\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.060038 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.060006 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" Sep 17 23:24:39.060472 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.060451 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" Sep 17 23:24:39.114263 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.114174 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-istio-data\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.114744 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.114718 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-istio-data\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.114894 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.114874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-workload-socket\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.115036 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.115022 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6009324f-0107-4cd3-9890-9fa7521de183-istio-token\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.115201 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.115175 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-workload-socket\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.115290 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.115191 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6009324f-0107-4cd3-9890-9fa7521de183-istio-podinfo\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.115468 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.115449 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhbx\" (UniqueName: \"kubernetes.io/projected/6009324f-0107-4cd3-9890-9fa7521de183-kube-api-access-zbhbx\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.115587 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.115571 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-credential-socket\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.115699 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.115684 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6009324f-0107-4cd3-9890-9fa7521de183-istiod-ca-cert\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.115844 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.115827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-workload-certs\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.115986 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.115972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-istio-envoy\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.116175 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.116150 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-workload-certs\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.116258 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.115916 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-credential-socket\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.116624 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.116592 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6009324f-0107-4cd3-9890-9fa7521de183-istiod-ca-cert\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.118865 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.118839 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6009324f-0107-4cd3-9890-9fa7521de183-istio-podinfo\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.119298 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.119273 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-istio-envoy\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.124181 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.124158 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6009324f-0107-4cd3-9890-9fa7521de183-istio-token\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.124743 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.124716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhbx\" (UniqueName: \"kubernetes.io/projected/6009324f-0107-4cd3-9890-9fa7521de183-kube-api-access-zbhbx\") pod \"e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.140248 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.140229 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:39.422694 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.422606 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d"] Sep 17 23:24:39.426186 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:24:39.426156 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6009324f_0107_4cd3_9890_9fa7521de183.slice/crio-9190b8d16c6e5c9f9eff1796e1af8347bef455bdf533fce4c7ff0d08c20cac16 WatchSource:0}: Error finding container 9190b8d16c6e5c9f9eff1796e1af8347bef455bdf533fce4c7ff0d08c20cac16: Status 404 returned error can't find the container with id 9190b8d16c6e5c9f9eff1796e1af8347bef455bdf533fce4c7ff0d08c20cac16 Sep 17 23:24:39.428407 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.428373 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:24:39.428502 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.428439 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:24:39.428502 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:39.428467 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:24:40.122362 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:40.122327 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" event={"ID":"6009324f-0107-4cd3-9890-9fa7521de183","Type":"ContainerStarted","Data":"745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74"} Sep 17 23:24:40.122749 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:40.122383 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" event={"ID":"6009324f-0107-4cd3-9890-9fa7521de183","Type":"ContainerStarted","Data":"9190b8d16c6e5c9f9eff1796e1af8347bef455bdf533fce4c7ff0d08c20cac16"} Sep 17 23:24:40.140860 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:40.140831 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:40.146841 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:40.146811 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:40.147033 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:40.146984 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" podStartSLOduration=2.14696484 podStartE2EDuration="2.14696484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:40.144930136 +0000 UTC m=+1108.262409611" watchObservedRunningTime="2026-09-17 23:24:40.14696484 +0000 UTC m=+1108.264444312" Sep 17 23:24:41.127988 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:41.127957 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:41.128886 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:41.128868 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:24:43.357824 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.357770 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:24:43.455063 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455030 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d40266b-e47c-4c17-8245-e25b227aa17a-istiod-ca-cert\") pod \"3d40266b-e47c-4c17-8245-e25b227aa17a\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " Sep 17 23:24:43.455243 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455068 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-token\") pod \"3d40266b-e47c-4c17-8245-e25b227aa17a\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " Sep 17 23:24:43.455243 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455123 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqv8r\" (UniqueName: \"kubernetes.io/projected/3d40266b-e47c-4c17-8245-e25b227aa17a-kube-api-access-dqv8r\") pod \"3d40266b-e47c-4c17-8245-e25b227aa17a\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " Sep 17 23:24:43.455243 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455144 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-podinfo\") pod \"3d40266b-e47c-4c17-8245-e25b227aa17a\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " Sep 17 23:24:43.455243 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455170 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-credential-socket\") pod \"3d40266b-e47c-4c17-8245-e25b227aa17a\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " Sep 17 23:24:43.455243 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455198 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-workload-certs\") pod \"3d40266b-e47c-4c17-8245-e25b227aa17a\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " Sep 17 23:24:43.455504 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455248 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-envoy\") pod \"3d40266b-e47c-4c17-8245-e25b227aa17a\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " Sep 17 23:24:43.455504 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455397 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d40266b-e47c-4c17-8245-e25b227aa17a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d40266b-e47c-4c17-8245-e25b227aa17a" (UID: "3d40266b-e47c-4c17-8245-e25b227aa17a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:24:43.455609 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455524 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d40266b-e47c-4c17-8245-e25b227aa17a" (UID: "3d40266b-e47c-4c17-8245-e25b227aa17a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:24:43.455609 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455541 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d40266b-e47c-4c17-8245-e25b227aa17a" (UID: "3d40266b-e47c-4c17-8245-e25b227aa17a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:24:43.455609 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455598 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-data\") pod \"3d40266b-e47c-4c17-8245-e25b227aa17a\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " Sep 17 23:24:43.455755 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455686 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-workload-socket\") pod \"3d40266b-e47c-4c17-8245-e25b227aa17a\" (UID: \"3d40266b-e47c-4c17-8245-e25b227aa17a\") " Sep 17 23:24:43.455855 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455815 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d40266b-e47c-4c17-8245-e25b227aa17a" (UID: "3d40266b-e47c-4c17-8245-e25b227aa17a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:24:43.456020 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.455921 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d40266b-e47c-4c17-8245-e25b227aa17a" (UID: "3d40266b-e47c-4c17-8245-e25b227aa17a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:24:43.456169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.456144 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:43.456169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.456173 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d40266b-e47c-4c17-8245-e25b227aa17a-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:43.456331 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.456188 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:43.456331 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.456201 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:43.456331 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.456214 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:43.457390 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.457363 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d40266b-e47c-4c17-8245-e25b227aa17a-kube-api-access-dqv8r" (OuterVolumeSpecName: "kube-api-access-dqv8r") pod "3d40266b-e47c-4c17-8245-e25b227aa17a" (UID: "3d40266b-e47c-4c17-8245-e25b227aa17a"). InnerVolumeSpecName "kube-api-access-dqv8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:24:43.457568 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.457547 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d40266b-e47c-4c17-8245-e25b227aa17a" (UID: "3d40266b-e47c-4c17-8245-e25b227aa17a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:24:43.457670 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.457654 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d40266b-e47c-4c17-8245-e25b227aa17a" (UID: "3d40266b-e47c-4c17-8245-e25b227aa17a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:24:43.457708 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.457675 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d40266b-e47c-4c17-8245-e25b227aa17a" (UID: "3d40266b-e47c-4c17-8245-e25b227aa17a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:24:43.556871 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.556836 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqv8r\" (UniqueName: \"kubernetes.io/projected/3d40266b-e47c-4c17-8245-e25b227aa17a-kube-api-access-dqv8r\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:43.556871 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.556866 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:43.556871 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.556878 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:43.557112 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:43.556890 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d40266b-e47c-4c17-8245-e25b227aa17a-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:24:44.159110 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.159075 2559 generic.go:356] "Generic (PLEG): container finished" podID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerID="80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484" exitCode=0 Sep 17 23:24:44.159303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.159150 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" Sep 17 23:24:44.159303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.159165 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" event={"ID":"3d40266b-e47c-4c17-8245-e25b227aa17a","Type":"ContainerDied","Data":"80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484"} Sep 17 23:24:44.159303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.159268 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz" event={"ID":"3d40266b-e47c-4c17-8245-e25b227aa17a","Type":"ContainerDied","Data":"9d25886c63573eb35af8566fad5f1ce75157bbad910190e700f119332f609f70"} Sep 17 23:24:44.159303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.159302 2559 scope.go:117] "RemoveContainer" containerID="80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484" Sep 17 23:24:44.169039 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.169022 2559 scope.go:117] "RemoveContainer" containerID="639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93" Sep 17 23:24:44.176911 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.176887 2559 scope.go:117] "RemoveContainer" containerID="80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484" Sep 17 23:24:44.177176 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:24:44.177158 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484\": container with ID starting with 80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484 not found: ID does not exist" containerID="80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484" Sep 17 23:24:44.177227 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.177184 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484"} err="failed to get container status \"80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484\": rpc error: code = NotFound desc = could not find container \"80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484\": container with ID starting with 80ad5d50184339738d180d6e5bdaed0a4abfb6adef38998d02ab43e3e97ea484 not found: ID does not exist" Sep 17 23:24:44.177227 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.177200 2559 scope.go:117] "RemoveContainer" containerID="639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93" Sep 17 23:24:44.177447 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:24:44.177431 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93\": container with ID starting with 639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93 not found: ID does not exist" containerID="639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93" Sep 17 23:24:44.177492 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.177453 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93"} err="failed to get container status \"639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93\": rpc error: code = NotFound desc = could not find container \"639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93\": container with ID starting with 639fc1b99df72f29832bb90dd9db45ed73206e2c1feba74a738c80eb9f78fa93 not found: ID does not exist" Sep 17 23:24:44.181173 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.181153 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz"] Sep 17 23:24:44.185770 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.185752 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-35df6d-openshift-default-6cccb6d8dd-jq9dz"] Sep 17 23:24:44.287814 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:44.287787 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" path="/var/lib/kubelet/pods/3d40266b-e47c-4c17-8245-e25b227aa17a/volumes" Sep 17 23:24:47.299864 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:47.299830 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:24:52.506657 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.506625 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc"] Sep 17 23:24:52.507273 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.507257 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerName="istio-proxy" Sep 17 23:24:52.507329 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.507277 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerName="istio-proxy" Sep 17 23:24:52.507411 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.507399 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerName="istio-proxy" Sep 17 23:24:52.513066 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.513041 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.516231 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.516206 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-efffde-openshift-default-dockercfg-nvcml\"" Sep 17 23:24:52.522495 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.522472 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc"] Sep 17 23:24:52.525317 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.525293 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-envoy\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.525411 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.525326 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bba8ad1-648f-42ec-a7de-bb2673058f29-istiod-ca-cert\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.525411 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.525357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-data\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.525527 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.525434 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-token\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.525527 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.525474 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-workload-certs\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.525527 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.525511 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-credential-socket\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.525673 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.525548 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-workload-socket\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.525673 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.525586 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8b9p\" (UniqueName: \"kubernetes.io/projected/6bba8ad1-648f-42ec-a7de-bb2673058f29-kube-api-access-k8b9p\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.525673 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.525651 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-podinfo\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.626105 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.626071 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-data\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.626302 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.626113 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-token\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.626302 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.626139 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-workload-certs\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.626302 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.626167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-credential-socket\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.626302 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.626200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-workload-socket\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.626302 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.626230 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8b9p\" (UniqueName: \"kubernetes.io/projected/6bba8ad1-648f-42ec-a7de-bb2673058f29-kube-api-access-k8b9p\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.626302 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.626285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-podinfo\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.626615 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.626336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-envoy\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.626615 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.626365 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bba8ad1-648f-42ec-a7de-bb2673058f29-istiod-ca-cert\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.626615 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.626433 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-data\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.626764 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.626641 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-workload-socket\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.627143 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.627121 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-workload-certs\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.627262 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.627137 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bba8ad1-648f-42ec-a7de-bb2673058f29-istiod-ca-cert\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.627262 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.627167 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-credential-socket\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.629016 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.628996 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-envoy\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.629375 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.629355 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-podinfo\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.635045 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.635021 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8b9p\" (UniqueName: \"kubernetes.io/projected/6bba8ad1-648f-42ec-a7de-bb2673058f29-kube-api-access-k8b9p\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.635501 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.635482 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-token\") pod \"e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:52.826089 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:52.826009 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:53.126112 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:53.126085 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc"] Sep 17 23:24:53.127727 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:24:53.127699 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bba8ad1_648f_42ec_a7de_bb2673058f29.slice/crio-910f4ce5f9259c14bf91e81053539bfb02075d2fb3c0e85ed74ef8f7df0e336f WatchSource:0}: Error finding container 910f4ce5f9259c14bf91e81053539bfb02075d2fb3c0e85ed74ef8f7df0e336f: Status 404 returned error can't find the container with id 910f4ce5f9259c14bf91e81053539bfb02075d2fb3c0e85ed74ef8f7df0e336f Sep 17 23:24:53.130043 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:53.130008 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:24:53.130149 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:53.130075 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:24:53.130149 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:53.130102 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:24:53.210837 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:53.210811 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" event={"ID":"6bba8ad1-648f-42ec-a7de-bb2673058f29","Type":"ContainerStarted","Data":"910f4ce5f9259c14bf91e81053539bfb02075d2fb3c0e85ed74ef8f7df0e336f"} Sep 17 23:24:54.217759 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:54.217722 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" event={"ID":"6bba8ad1-648f-42ec-a7de-bb2673058f29","Type":"ContainerStarted","Data":"2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2"} Sep 17 23:24:54.241097 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:54.241039 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" podStartSLOduration=2.241018711 podStartE2EDuration="2.241018711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:54.237344581 +0000 UTC m=+1122.354824029" watchObservedRunningTime="2026-09-17 23:24:54.241018711 +0000 UTC m=+1122.358498163" Sep 17 23:24:54.826493 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:54.826456 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:54.833780 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:54.833755 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:55.224364 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:55.224327 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:55.225320 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:55.225295 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:24:58.243358 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:58.243327 2559 generic.go:356] "Generic (PLEG): container finished" podID="59837b90-41e0-4b62-a77f-5b100840f529" containerID="0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247" exitCode=0 Sep 17 23:24:58.243880 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:58.243388 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" event={"ID":"59837b90-41e0-4b62-a77f-5b100840f529","Type":"ContainerDied","Data":"0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247"} Sep 17 23:24:58.244003 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:58.243934 2559 scope.go:117] "RemoveContainer" containerID="0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247" Sep 17 23:24:58.245312 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:58.245283 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:24:58.245395 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:58.245353 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:24:58.245437 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:58.245389 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:24:59.257751 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:59.257705 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" event={"ID":"59837b90-41e0-4b62-a77f-5b100840f529","Type":"ContainerStarted","Data":"a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083"} Sep 17 23:24:59.358625 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:59.358578 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:24:59.358837 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:59.358634 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 17 23:24:59.358837 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:24:59.358688 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 17 23:25:00.358395 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.357957 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 17 23:25:00.358395 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.358031 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 17 23:25:00.421485 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.421438 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm"] Sep 17 23:25:00.422858 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.422834 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerName="istio-proxy" Sep 17 23:25:00.423022 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.423009 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerName="istio-proxy" Sep 17 23:25:00.423348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.423334 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d40266b-e47c-4c17-8245-e25b227aa17a" containerName="istio-proxy" Sep 17 23:25:00.427284 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.427262 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:00.432303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.432282 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-efffde-serving-cert\"" Sep 17 23:25:00.434149 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.433341 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 23:25:00.434149 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.433541 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 23:25:00.434149 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.433793 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 23:25:00.434149 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.434072 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cqmcq\"" Sep 17 23:25:00.437117 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.437032 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm"] Sep 17 23:25:00.492687 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.492647 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tw5f\" (UniqueName: \"kubernetes.io/projected/1268244f-5d3c-45b4-911a-621268865ccf-kube-api-access-9tw5f\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:00.492871 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.492704 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:00.492871 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.492831 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1268244f-5d3c-45b4-911a-621268865ccf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:00.493016 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.492931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:00.594419 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.594373 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1268244f-5d3c-45b4-911a-621268865ccf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:00.594614 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.594435 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:00.594614 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.594491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tw5f\" (UniqueName: \"kubernetes.io/projected/1268244f-5d3c-45b4-911a-621268865ccf-kube-api-access-9tw5f\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:00.594614 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.594511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:00.594614 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:00.594607 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-serving-cert: secret "maas-api-e2e-shared-b-w2-efffde-serving-cert" not found Sep 17 23:25:00.594822 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:00.594666 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-maas-api-tls podName:1268244f-5d3c-45b4-911a-621268865ccf nodeName:}" failed. No retries permitted until 2026-09-17 23:25:01.094647417 +0000 UTC m=+1129.212126867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-maas-api-tls") pod "maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" (UID: "1268244f-5d3c-45b4-911a-621268865ccf") : secret "maas-api-e2e-shared-b-w2-efffde-serving-cert" not found Sep 17 23:25:00.595156 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.595131 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1268244f-5d3c-45b4-911a-621268865ccf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:00.597041 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.597014 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:00.604590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:00.604567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tw5f\" (UniqueName: \"kubernetes.io/projected/1268244f-5d3c-45b4-911a-621268865ccf-kube-api-access-9tw5f\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:01.066600 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.066548 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7"] Sep 17 23:25:01.071154 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.071127 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" Sep 17 23:25:01.073812 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.073789 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-efffde\"" Sep 17 23:25:01.080347 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.080322 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7"] Sep 17 23:25:01.100072 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.100044 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0356abbb-3e8e-41fb-bace-a1375b4cd869-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7\" (UID: \"0356abbb-3e8e-41fb-bace-a1375b4cd869\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" Sep 17 23:25:01.100189 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.100081 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b77n\" (UniqueName: \"kubernetes.io/projected/0356abbb-3e8e-41fb-bace-a1375b4cd869-kube-api-access-4b77n\") pod \"payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7\" (UID: \"0356abbb-3e8e-41fb-bace-a1375b4cd869\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" Sep 17 23:25:01.100189 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.100111 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:01.102701 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.102677 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:01.200725 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.200688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b77n\" (UniqueName: \"kubernetes.io/projected/0356abbb-3e8e-41fb-bace-a1375b4cd869-kube-api-access-4b77n\") pod \"payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7\" (UID: \"0356abbb-3e8e-41fb-bace-a1375b4cd869\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" Sep 17 23:25:01.200924 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.200784 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0356abbb-3e8e-41fb-bace-a1375b4cd869-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7\" (UID: \"0356abbb-3e8e-41fb-bace-a1375b4cd869\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" Sep 17 23:25:01.201571 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.201544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0356abbb-3e8e-41fb-bace-a1375b4cd869-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7\" (UID: \"0356abbb-3e8e-41fb-bace-a1375b4cd869\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" Sep 17 23:25:01.210146 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.210125 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b77n\" (UniqueName: \"kubernetes.io/projected/0356abbb-3e8e-41fb-bace-a1375b4cd869-kube-api-access-4b77n\") pod \"payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7\" (UID: \"0356abbb-3e8e-41fb-bace-a1375b4cd869\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" Sep 17 23:25:01.340168 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.340073 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:01.363925 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.361860 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 17 23:25:01.363925 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.361940 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 17 23:25:01.383244 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.383206 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" Sep 17 23:25:01.635073 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.635045 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm"] Sep 17 23:25:01.636526 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:25:01.636502 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1268244f_5d3c_45b4_911a_621268865ccf.slice/crio-86ff67f25e1b1f561a4d4ef3fdc622e7f33e17e24f50f9d104e87b285d6168b8 WatchSource:0}: Error finding container 86ff67f25e1b1f561a4d4ef3fdc622e7f33e17e24f50f9d104e87b285d6168b8: Status 404 returned error can't find the container with id 86ff67f25e1b1f561a4d4ef3fdc622e7f33e17e24f50f9d104e87b285d6168b8 Sep 17 23:25:01.637992 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.637973 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 23:25:01.701233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:01.701210 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7"] Sep 17 23:25:01.703085 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:25:01.703055 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0356abbb_3e8e_41fb_bace_a1375b4cd869.slice/crio-a3f841d53133f0f7637428205f9b4cb2539bd04f7574a988af77529f7de932f3 WatchSource:0}: Error finding container a3f841d53133f0f7637428205f9b4cb2539bd04f7574a988af77529f7de932f3: Status 404 returned error can't find the container with id a3f841d53133f0f7637428205f9b4cb2539bd04f7574a988af77529f7de932f3 Sep 17 23:25:02.286967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:02.286874 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" event={"ID":"0356abbb-3e8e-41fb-bace-a1375b4cd869","Type":"ContainerStarted","Data":"ef333522a2752632a4832ad3f230ac1731641bdf049d78997904712da81f3d2b"} Sep 17 23:25:02.286967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:02.286941 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" Sep 17 23:25:02.286967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:02.286968 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" event={"ID":"0356abbb-3e8e-41fb-bace-a1375b4cd869","Type":"ContainerStarted","Data":"a3f841d53133f0f7637428205f9b4cb2539bd04f7574a988af77529f7de932f3"} Sep 17 23:25:02.287188 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:02.286982 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" event={"ID":"1268244f-5d3c-45b4-911a-621268865ccf","Type":"ContainerStarted","Data":"f5c2c43ea075ba8d51d87ffcf5b0b45c7e5fa477ac4c1033d213d1fa946bcb90"} Sep 17 23:25:02.287188 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:02.286998 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" event={"ID":"1268244f-5d3c-45b4-911a-621268865ccf","Type":"ContainerStarted","Data":"86ff67f25e1b1f561a4d4ef3fdc622e7f33e17e24f50f9d104e87b285d6168b8"} Sep 17 23:25:02.287188 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:02.287012 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:02.298247 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:02.298203 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" podStartSLOduration=1.2981859660000001 podStartE2EDuration="1.298185966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:02.295629095 +0000 UTC m=+1130.413108573" watchObservedRunningTime="2026-09-17 23:25:02.298185966 +0000 UTC m=+1130.415665415" Sep 17 23:25:02.314731 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:02.314687 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" podStartSLOduration=1.986193696 podStartE2EDuration="2.314673875s" totalImagesPullingTime="328.480179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:02.313685473 +0000 UTC m=+1130.431164941" watchObservedRunningTime="2026-09-17 23:25:02.314673875 +0000 UTC m=+1130.432153325" Sep 17 23:25:02.357965 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:02.357928 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 17 23:25:02.358210 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:02.358179 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 17 23:25:03.358629 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:03.358586 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 17 23:25:03.359103 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:03.358651 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 17 23:25:04.361679 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:04.361646 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:25:04.362076 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:04.362053 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:25:04.362665 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:04.362637 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:25:08.293765 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:08.293734 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:25:13.290535 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:13.290506 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" Sep 17 23:25:21.409240 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:21.409207 2559 generic.go:356] "Generic (PLEG): container finished" podID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerID="2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2" exitCode=0 Sep 17 23:25:21.409650 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:21.409272 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" event={"ID":"6bba8ad1-648f-42ec-a7de-bb2673058f29","Type":"ContainerDied","Data":"2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2"} Sep 17 23:25:21.409650 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:21.409580 2559 scope.go:117] "RemoveContainer" containerID="2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2" Sep 17 23:25:21.410588 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:21.410561 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:25:21.410654 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:21.410623 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:25:21.410689 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:21.410654 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:25:22.417045 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:22.417010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" event={"ID":"6bba8ad1-648f-42ec-a7de-bb2673058f29","Type":"ContainerStarted","Data":"844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd"} Sep 17 23:25:22.826620 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:22.826584 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:25:22.827857 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:22.827832 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 17 23:25:22.828014 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:22.827882 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 17 23:25:23.826822 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:23.826783 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 17 23:25:23.827243 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:23.826851 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 17 23:25:24.827049 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:24.827007 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 17 23:25:24.827534 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:24.827096 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 17 23:25:25.224261 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:25.224228 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:25:25.826890 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:25.826851 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 17 23:25:25.827116 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:25.826934 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 17 23:25:26.826506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:26.826465 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 17 23:25:26.826679 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:26.826537 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 17 23:25:26.933121 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:26.933086 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7850-9dba10"] Sep 17 23:25:26.937035 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:26.937018 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-9dba10" Sep 17 23:25:26.941318 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:26.941289 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pml6m\"" Sep 17 23:25:26.953661 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:26.953630 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7850-9dba10"] Sep 17 23:25:27.055458 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:27.055421 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2knk\" (UniqueName: \"kubernetes.io/projected/88ffb21e-47b4-4eea-845d-d50e79fc696c-kube-api-access-q2knk\") pod \"test-curl-7850-9dba10\" (UID: \"88ffb21e-47b4-4eea-845d-d50e79fc696c\") " pod="opendatahub/test-curl-7850-9dba10" Sep 17 23:25:27.157310 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:27.157224 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2knk\" (UniqueName: \"kubernetes.io/projected/88ffb21e-47b4-4eea-845d-d50e79fc696c-kube-api-access-q2knk\") pod \"test-curl-7850-9dba10\" (UID: \"88ffb21e-47b4-4eea-845d-d50e79fc696c\") " pod="opendatahub/test-curl-7850-9dba10" Sep 17 23:25:27.167591 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:27.167566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2knk\" (UniqueName: \"kubernetes.io/projected/88ffb21e-47b4-4eea-845d-d50e79fc696c-kube-api-access-q2knk\") pod \"test-curl-7850-9dba10\" (UID: \"88ffb21e-47b4-4eea-845d-d50e79fc696c\") " pod="opendatahub/test-curl-7850-9dba10" Sep 17 23:25:27.249701 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:27.249669 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-9dba10" Sep 17 23:25:27.532132 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:27.532097 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7850-9dba10"] Sep 17 23:25:27.539231 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:25:27.539203 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88ffb21e_47b4_4eea_845d_d50e79fc696c.slice/crio-bce2075b49978d2d1044a469801864948d60c480251195cc89223e7c5a79a40c WatchSource:0}: Error finding container bce2075b49978d2d1044a469801864948d60c480251195cc89223e7c5a79a40c: Status 404 returned error can't find the container with id bce2075b49978d2d1044a469801864948d60c480251195cc89223e7c5a79a40c Sep 17 23:25:27.831336 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:27.831246 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:25:27.832108 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:27.832091 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:25:28.460077 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:28.460025 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7850-9dba10" event={"ID":"88ffb21e-47b4-4eea-845d-d50e79fc696c","Type":"ContainerStarted","Data":"54f2d853a6db83553e57e29daee3e56021a0168fc5ba20b757df2ffd6f66f959"} Sep 17 23:25:28.460506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:28.460084 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7850-9dba10" event={"ID":"88ffb21e-47b4-4eea-845d-d50e79fc696c","Type":"ContainerStarted","Data":"bce2075b49978d2d1044a469801864948d60c480251195cc89223e7c5a79a40c"} Sep 17 23:25:28.478331 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:28.478260 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7850-9dba10" podStartSLOduration=2.478240693 podStartE2EDuration="2.478240693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:28.475678724 +0000 UTC m=+1156.593158175" watchObservedRunningTime="2026-09-17 23:25:28.478240693 +0000 UTC m=+1156.595720145" Sep 17 23:25:29.052936 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.052891 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7850-9dba10"] Sep 17 23:25:29.057996 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.057967 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7850-9dba10"] Sep 17 23:25:29.172874 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.172801 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7850-2dd76d"] Sep 17 23:25:29.173279 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.173266 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ffb21e-47b4-4eea-845d-d50e79fc696c" containerName="test-curl-7850-9dba10" Sep 17 23:25:29.173325 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.173280 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ffb21e-47b4-4eea-845d-d50e79fc696c" containerName="test-curl-7850-9dba10" Sep 17 23:25:29.173367 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.173358 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ffb21e-47b4-4eea-845d-d50e79fc696c" containerName="test-curl-7850-9dba10" Sep 17 23:25:29.176594 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.176569 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-2dd76d" Sep 17 23:25:29.183075 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.183053 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7850-2dd76d"] Sep 17 23:25:29.280554 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.280522 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4tl5\" (UniqueName: \"kubernetes.io/projected/4d27a597-1450-4877-9251-ee619cb02973-kube-api-access-b4tl5\") pod \"test-curl-7850-2dd76d\" (UID: \"4d27a597-1450-4877-9251-ee619cb02973\") " pod="opendatahub/test-curl-7850-2dd76d" Sep 17 23:25:29.381656 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.381625 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4tl5\" (UniqueName: \"kubernetes.io/projected/4d27a597-1450-4877-9251-ee619cb02973-kube-api-access-b4tl5\") pod \"test-curl-7850-2dd76d\" (UID: \"4d27a597-1450-4877-9251-ee619cb02973\") " pod="opendatahub/test-curl-7850-2dd76d" Sep 17 23:25:29.389707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.389682 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4tl5\" (UniqueName: \"kubernetes.io/projected/4d27a597-1450-4877-9251-ee619cb02973-kube-api-access-b4tl5\") pod \"test-curl-7850-2dd76d\" (UID: \"4d27a597-1450-4877-9251-ee619cb02973\") " pod="opendatahub/test-curl-7850-2dd76d" Sep 17 23:25:29.464891 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:29.464863 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7850-9dba10" container="test-curl-7850-9dba10" Sep 17 23:25:29.488028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.488005 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-2dd76d" Sep 17 23:25:29.756089 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:29.756063 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7850-2dd76d"] Sep 17 23:25:29.758069 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:25:29.758041 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d27a597_1450_4877_9251_ee619cb02973.slice/crio-ac969d7cdf212866ddcfd3b6ebf8752d497771c66e9f96c8d84c33174600c1b6 WatchSource:0}: Error finding container ac969d7cdf212866ddcfd3b6ebf8752d497771c66e9f96c8d84c33174600c1b6: Status 404 returned error can't find the container with id ac969d7cdf212866ddcfd3b6ebf8752d497771c66e9f96c8d84c33174600c1b6 Sep 17 23:25:30.471803 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:30.471765 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7850-2dd76d" event={"ID":"4d27a597-1450-4877-9251-ee619cb02973","Type":"ContainerStarted","Data":"aa358cdb889c26f3d8360244ccadffd5e29208b3609a3919726f31cc8c2fc025"} Sep 17 23:25:30.471803 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:30.471808 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7850-2dd76d" event={"ID":"4d27a597-1450-4877-9251-ee619cb02973","Type":"ContainerStarted","Data":"ac969d7cdf212866ddcfd3b6ebf8752d497771c66e9f96c8d84c33174600c1b6"} Sep 17 23:25:30.472292 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:30.472000 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7850-9dba10" podUID="88ffb21e-47b4-4eea-845d-d50e79fc696c" containerName="test-curl-7850-9dba10" containerID="cri-o://54f2d853a6db83553e57e29daee3e56021a0168fc5ba20b757df2ffd6f66f959" gracePeriod=2 Sep 17 23:25:30.489779 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:30.489733 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7850-2dd76d" podStartSLOduration=1.489719452 podStartE2EDuration="1.489719452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:30.487526478 +0000 UTC m=+1158.605005931" watchObservedRunningTime="2026-09-17 23:25:30.489719452 +0000 UTC m=+1158.607198902" Sep 17 23:25:31.083669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.083632 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7850-2dd76d"] Sep 17 23:25:31.087245 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.087218 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7850-2dd76d"] Sep 17 23:25:31.195790 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.195722 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7850-57ba95"] Sep 17 23:25:31.196236 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.196222 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d27a597-1450-4877-9251-ee619cb02973" containerName="test-curl-7850-2dd76d" Sep 17 23:25:31.196283 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.196238 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d27a597-1450-4877-9251-ee619cb02973" containerName="test-curl-7850-2dd76d" Sep 17 23:25:31.196334 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.196325 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d27a597-1450-4877-9251-ee619cb02973" containerName="test-curl-7850-2dd76d" Sep 17 23:25:31.199303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.199288 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-57ba95" Sep 17 23:25:31.209669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.209639 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7850-57ba95"] Sep 17 23:25:31.296229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.296200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftn74\" (UniqueName: \"kubernetes.io/projected/bbeae5da-78fc-4ae3-9824-97f4ad4b367a-kube-api-access-ftn74\") pod \"test-curl-7850-57ba95\" (UID: \"bbeae5da-78fc-4ae3-9824-97f4ad4b367a\") " pod="opendatahub/test-curl-7850-57ba95" Sep 17 23:25:31.397675 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.397636 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftn74\" (UniqueName: \"kubernetes.io/projected/bbeae5da-78fc-4ae3-9824-97f4ad4b367a-kube-api-access-ftn74\") pod \"test-curl-7850-57ba95\" (UID: \"bbeae5da-78fc-4ae3-9824-97f4ad4b367a\") " pod="opendatahub/test-curl-7850-57ba95" Sep 17 23:25:31.406963 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.406937 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftn74\" (UniqueName: \"kubernetes.io/projected/bbeae5da-78fc-4ae3-9824-97f4ad4b367a-kube-api-access-ftn74\") pod \"test-curl-7850-57ba95\" (UID: \"bbeae5da-78fc-4ae3-9824-97f4ad4b367a\") " pod="opendatahub/test-curl-7850-57ba95" Sep 17 23:25:31.477881 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:31.477855 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7850-2dd76d" container="test-curl-7850-2dd76d" Sep 17 23:25:31.508826 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.508805 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-57ba95" Sep 17 23:25:31.783551 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:31.783513 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7850-57ba95"] Sep 17 23:25:31.785789 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:25:31.785761 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbeae5da_78fc_4ae3_9824_97f4ad4b367a.slice/crio-fb7d352341f9b0430b09a4a4b1552276bfd9bc565d5a140c94fb3f2d5ee34384 WatchSource:0}: Error finding container fb7d352341f9b0430b09a4a4b1552276bfd9bc565d5a140c94fb3f2d5ee34384: Status 404 returned error can't find the container with id fb7d352341f9b0430b09a4a4b1552276bfd9bc565d5a140c94fb3f2d5ee34384 Sep 17 23:25:32.486072 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:32.486034 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7850-57ba95" event={"ID":"bbeae5da-78fc-4ae3-9824-97f4ad4b367a","Type":"ContainerStarted","Data":"70226c75a9de2cb85bafca12059c4ac35ea2c03bae38449bf11786a7c8025779"} Sep 17 23:25:32.486443 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:32.486088 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7850-57ba95" event={"ID":"bbeae5da-78fc-4ae3-9824-97f4ad4b367a","Type":"ContainerStarted","Data":"fb7d352341f9b0430b09a4a4b1552276bfd9bc565d5a140c94fb3f2d5ee34384"} Sep 17 23:25:32.486443 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:32.486080 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7850-2dd76d" podUID="4d27a597-1450-4877-9251-ee619cb02973" containerName="test-curl-7850-2dd76d" containerID="cri-o://aa358cdb889c26f3d8360244ccadffd5e29208b3609a3919726f31cc8c2fc025" gracePeriod=2 Sep 17 23:25:32.503914 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:32.503878 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:25:32.505185 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:32.505144 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7850-57ba95" podStartSLOduration=1.505129334 podStartE2EDuration="1.505129334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:32.503613052 +0000 UTC m=+1160.621092502" watchObservedRunningTime="2026-09-17 23:25:32.505129334 +0000 UTC m=+1160.622608784" Sep 17 23:25:32.658236 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:32.658212 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-9dba10" Sep 17 23:25:32.811142 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:32.811061 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2knk\" (UniqueName: \"kubernetes.io/projected/88ffb21e-47b4-4eea-845d-d50e79fc696c-kube-api-access-q2knk\") pod \"88ffb21e-47b4-4eea-845d-d50e79fc696c\" (UID: \"88ffb21e-47b4-4eea-845d-d50e79fc696c\") " Sep 17 23:25:32.813073 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:32.813045 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ffb21e-47b4-4eea-845d-d50e79fc696c-kube-api-access-q2knk" (OuterVolumeSpecName: "kube-api-access-q2knk") pod "88ffb21e-47b4-4eea-845d-d50e79fc696c" (UID: "88ffb21e-47b4-4eea-845d-d50e79fc696c"). InnerVolumeSpecName "kube-api-access-q2knk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:32.912094 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:32.912058 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2knk\" (UniqueName: \"kubernetes.io/projected/88ffb21e-47b4-4eea-845d-d50e79fc696c-kube-api-access-q2knk\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:25:33.087819 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.087719 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7850-57ba95"] Sep 17 23:25:33.095137 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.095105 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7850-57ba95"] Sep 17 23:25:33.203405 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.203375 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7850-b3181f"] Sep 17 23:25:33.203829 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.203817 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbeae5da-78fc-4ae3-9824-97f4ad4b367a" containerName="test-curl-7850-57ba95" Sep 17 23:25:33.203874 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.203831 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbeae5da-78fc-4ae3-9824-97f4ad4b367a" containerName="test-curl-7850-57ba95" Sep 17 23:25:33.203918 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.203894 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbeae5da-78fc-4ae3-9824-97f4ad4b367a" containerName="test-curl-7850-57ba95" Sep 17 23:25:33.207283 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.207262 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-b3181f" Sep 17 23:25:33.212233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.212205 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7850-b3181f"] Sep 17 23:25:33.316791 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.316758 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-677w4\" (UniqueName: \"kubernetes.io/projected/d25533bf-a2af-4951-8a5b-a932ae379a39-kube-api-access-677w4\") pod \"test-curl-7850-b3181f\" (UID: \"d25533bf-a2af-4951-8a5b-a932ae379a39\") " pod="opendatahub/test-curl-7850-b3181f" Sep 17 23:25:33.418007 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.417932 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-677w4\" (UniqueName: \"kubernetes.io/projected/d25533bf-a2af-4951-8a5b-a932ae379a39-kube-api-access-677w4\") pod \"test-curl-7850-b3181f\" (UID: \"d25533bf-a2af-4951-8a5b-a932ae379a39\") " pod="opendatahub/test-curl-7850-b3181f" Sep 17 23:25:33.426676 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.426648 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-677w4\" (UniqueName: \"kubernetes.io/projected/d25533bf-a2af-4951-8a5b-a932ae379a39-kube-api-access-677w4\") pod \"test-curl-7850-b3181f\" (UID: \"d25533bf-a2af-4951-8a5b-a932ae379a39\") " pod="opendatahub/test-curl-7850-b3181f" Sep 17 23:25:33.492947 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.492920 2559 generic.go:356] "Generic (PLEG): container finished" podID="88ffb21e-47b4-4eea-845d-d50e79fc696c" containerID="54f2d853a6db83553e57e29daee3e56021a0168fc5ba20b757df2ffd6f66f959" exitCode=137 Sep 17 23:25:33.493304 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.492964 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-9dba10" Sep 17 23:25:33.493304 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.493013 2559 scope.go:117] "RemoveContainer" containerID="54f2d853a6db83553e57e29daee3e56021a0168fc5ba20b757df2ffd6f66f959" Sep 17 23:25:33.493407 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:33.493377 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7850-57ba95" container="test-curl-7850-57ba95" Sep 17 23:25:33.502097 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.502081 2559 scope.go:117] "RemoveContainer" containerID="54f2d853a6db83553e57e29daee3e56021a0168fc5ba20b757df2ffd6f66f959" Sep 17 23:25:33.502371 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:33.502353 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f2d853a6db83553e57e29daee3e56021a0168fc5ba20b757df2ffd6f66f959\": container with ID starting with 54f2d853a6db83553e57e29daee3e56021a0168fc5ba20b757df2ffd6f66f959 not found: ID does not exist" containerID="54f2d853a6db83553e57e29daee3e56021a0168fc5ba20b757df2ffd6f66f959" Sep 17 23:25:33.502425 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.502380 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f2d853a6db83553e57e29daee3e56021a0168fc5ba20b757df2ffd6f66f959"} err="failed to get container status \"54f2d853a6db83553e57e29daee3e56021a0168fc5ba20b757df2ffd6f66f959\": rpc error: code = NotFound desc = could not find container \"54f2d853a6db83553e57e29daee3e56021a0168fc5ba20b757df2ffd6f66f959\": container with ID starting with 54f2d853a6db83553e57e29daee3e56021a0168fc5ba20b757df2ffd6f66f959 not found: ID does not exist" Sep 17 23:25:33.503615 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:33.503597 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7850-9dba10\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="88ffb21e-47b4-4eea-845d-d50e79fc696c" pod="opendatahub/test-curl-7850-9dba10" Sep 17 23:25:33.518793 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.518772 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-b3181f" Sep 17 23:25:33.789706 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:33.789680 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7850-b3181f"] Sep 17 23:25:33.791052 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:25:33.791026 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd25533bf_a2af_4951_8a5b_a932ae379a39.slice/crio-e2ca2b1a9ac428b40e1c35ec5632e98339f13009bd15a8d057bbf5b8114f0f9f WatchSource:0}: Error finding container e2ca2b1a9ac428b40e1c35ec5632e98339f13009bd15a8d057bbf5b8114f0f9f: Status 404 returned error can't find the container with id e2ca2b1a9ac428b40e1c35ec5632e98339f13009bd15a8d057bbf5b8114f0f9f Sep 17 23:25:34.286458 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:34.286425 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7850-9dba10\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="88ffb21e-47b4-4eea-845d-d50e79fc696c" pod="opendatahub/test-curl-7850-9dba10" Sep 17 23:25:34.288176 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:34.288152 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88ffb21e-47b4-4eea-845d-d50e79fc696c" path="/var/lib/kubelet/pods/88ffb21e-47b4-4eea-845d-d50e79fc696c/volumes" Sep 17 23:25:34.502457 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:34.502431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7850-b3181f" event={"ID":"d25533bf-a2af-4951-8a5b-a932ae379a39","Type":"ContainerStarted","Data":"bd0ccaac46ddb473847cd3b7272a3dabdd1640c928aa34b45c93162fbac8b950"} Sep 17 23:25:34.502781 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:34.502473 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7850-b3181f" event={"ID":"d25533bf-a2af-4951-8a5b-a932ae379a39","Type":"ContainerStarted","Data":"e2ca2b1a9ac428b40e1c35ec5632e98339f13009bd15a8d057bbf5b8114f0f9f"} Sep 17 23:25:34.502781 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:34.502471 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7850-57ba95" podUID="bbeae5da-78fc-4ae3-9824-97f4ad4b367a" containerName="test-curl-7850-57ba95" containerID="cri-o://70226c75a9de2cb85bafca12059c4ac35ea2c03bae38449bf11786a7c8025779" gracePeriod=2 Sep 17 23:25:34.518041 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:34.517991 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7850-b3181f" podStartSLOduration=1.5179716650000001 podStartE2EDuration="1.517971665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:34.517623438 +0000 UTC m=+1162.635102891" watchObservedRunningTime="2026-09-17 23:25:34.517971665 +0000 UTC m=+1162.635451117" Sep 17 23:25:34.663177 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:34.663152 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-2dd76d" Sep 17 23:25:34.832262 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:34.832179 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4tl5\" (UniqueName: \"kubernetes.io/projected/4d27a597-1450-4877-9251-ee619cb02973-kube-api-access-b4tl5\") pod \"4d27a597-1450-4877-9251-ee619cb02973\" (UID: \"4d27a597-1450-4877-9251-ee619cb02973\") " Sep 17 23:25:34.834043 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:34.834019 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d27a597-1450-4877-9251-ee619cb02973-kube-api-access-b4tl5" (OuterVolumeSpecName: "kube-api-access-b4tl5") pod "4d27a597-1450-4877-9251-ee619cb02973" (UID: "4d27a597-1450-4877-9251-ee619cb02973"). InnerVolumeSpecName "kube-api-access-b4tl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:34.933506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:34.933480 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4tl5\" (UniqueName: \"kubernetes.io/projected/4d27a597-1450-4877-9251-ee619cb02973-kube-api-access-b4tl5\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:25:35.104911 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.104822 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7850-b3181f"] Sep 17 23:25:35.110681 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.110654 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7850-b3181f"] Sep 17 23:25:35.513658 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.513619 2559 generic.go:356] "Generic (PLEG): container finished" podID="4d27a597-1450-4877-9251-ee619cb02973" containerID="aa358cdb889c26f3d8360244ccadffd5e29208b3609a3919726f31cc8c2fc025" exitCode=137 Sep 17 23:25:35.514155 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.513670 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-2dd76d" Sep 17 23:25:35.514155 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.513710 2559 scope.go:117] "RemoveContainer" containerID="aa358cdb889c26f3d8360244ccadffd5e29208b3609a3919726f31cc8c2fc025" Sep 17 23:25:35.514155 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:35.514021 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7850-b3181f" container="test-curl-7850-b3181f" Sep 17 23:25:35.522852 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.522836 2559 scope.go:117] "RemoveContainer" containerID="aa358cdb889c26f3d8360244ccadffd5e29208b3609a3919726f31cc8c2fc025" Sep 17 23:25:35.523153 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:35.523134 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa358cdb889c26f3d8360244ccadffd5e29208b3609a3919726f31cc8c2fc025\": container with ID starting with aa358cdb889c26f3d8360244ccadffd5e29208b3609a3919726f31cc8c2fc025 not found: ID does not exist" containerID="aa358cdb889c26f3d8360244ccadffd5e29208b3609a3919726f31cc8c2fc025" Sep 17 23:25:35.523215 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.523163 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa358cdb889c26f3d8360244ccadffd5e29208b3609a3919726f31cc8c2fc025"} err="failed to get container status \"aa358cdb889c26f3d8360244ccadffd5e29208b3609a3919726f31cc8c2fc025\": rpc error: code = NotFound desc = could not find container \"aa358cdb889c26f3d8360244ccadffd5e29208b3609a3919726f31cc8c2fc025\": container with ID starting with aa358cdb889c26f3d8360244ccadffd5e29208b3609a3919726f31cc8c2fc025 not found: ID does not exist" Sep 17 23:25:35.523467 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:35.523446 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7850-2dd76d\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="4d27a597-1450-4877-9251-ee619cb02973" pod="opendatahub/test-curl-7850-2dd76d" Sep 17 23:25:35.962887 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.962858 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762"] Sep 17 23:25:35.963586 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.963561 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25533bf-a2af-4951-8a5b-a932ae379a39" containerName="test-curl-7850-b3181f" Sep 17 23:25:35.963586 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.963584 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25533bf-a2af-4951-8a5b-a932ae379a39" containerName="test-curl-7850-b3181f" Sep 17 23:25:35.963772 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.963705 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d25533bf-a2af-4951-8a5b-a932ae379a39" containerName="test-curl-7850-b3181f" Sep 17 23:25:35.968065 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.968046 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:35.970477 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:35.970441 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7850-2dd76d\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="4d27a597-1450-4877-9251-ee619cb02973" pod="opendatahub/test-curl-7850-2dd76d" Sep 17 23:25:35.970612 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.970590 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-09450b8d-openshift-default-dockercfg-cpl2n\"" Sep 17 23:25:35.979981 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:35.979957 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762"] Sep 17 23:25:36.146598 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.146562 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-data\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.146769 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.146607 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1768fa61-e9d1-4147-bddc-1a1c283335aa-istiod-ca-cert\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.146769 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.146644 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-workload-socket\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.146769 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.146696 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-token\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.146769 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.146723 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-envoy\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.146769 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.146764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-workload-certs\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.147060 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.146817 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnjp7\" (UniqueName: \"kubernetes.io/projected/1768fa61-e9d1-4147-bddc-1a1c283335aa-kube-api-access-fnjp7\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.147060 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.146917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-credential-socket\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.147060 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.147006 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-podinfo\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.247539 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.247462 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-data\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.247539 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.247494 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1768fa61-e9d1-4147-bddc-1a1c283335aa-istiod-ca-cert\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.247539 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.247513 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-workload-socket\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.247808 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.247544 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-token\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.247808 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.247560 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-envoy\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.247808 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.247598 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-workload-certs\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.247808 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.247629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnjp7\" (UniqueName: \"kubernetes.io/projected/1768fa61-e9d1-4147-bddc-1a1c283335aa-kube-api-access-fnjp7\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.247808 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.247677 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-credential-socket\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.247808 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.247763 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-podinfo\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.248118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.247941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-data\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.248236 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.248204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-workload-certs\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.248345 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.248297 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-workload-socket\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.248345 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.248312 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1768fa61-e9d1-4147-bddc-1a1c283335aa-istiod-ca-cert\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.248425 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.248345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-credential-socket\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.250307 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.250283 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-envoy\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.250657 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.250639 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-podinfo\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.257393 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.257365 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-token\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.258274 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.258252 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnjp7\" (UniqueName: \"kubernetes.io/projected/1768fa61-e9d1-4147-bddc-1a1c283335aa-kube-api-access-fnjp7\") pod \"e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.279745 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.279719 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:36.289990 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.289965 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d27a597-1450-4877-9251-ee619cb02973" path="/var/lib/kubelet/pods/4d27a597-1450-4877-9251-ee619cb02973/volumes" Sep 17 23:25:36.329987 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.329948 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j"] Sep 17 23:25:36.330559 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.330207 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" podUID="b9e66f44-da70-4361-a106-a2a731870f47" containerName="payload-processing" containerID="cri-o://2422201a987441049f8a459a888b41193f88ca7653f58a0a2a16071f67386366" gracePeriod=30 Sep 17 23:25:36.350664 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.350058 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx"] Sep 17 23:25:36.350664 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.350281 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" podUID="7be52dd4-470a-437c-9ad9-a749a110efe6" containerName="payload-pre-processing" containerID="cri-o://72cfc9c38283e479650595986f226ca84f9dbf793b8da467d72c98ebee180055" gracePeriod=30 Sep 17 23:25:36.525057 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.524760 2559 generic.go:356] "Generic (PLEG): container finished" podID="b9e66f44-da70-4361-a106-a2a731870f47" containerID="2422201a987441049f8a459a888b41193f88ca7653f58a0a2a16071f67386366" exitCode=0 Sep 17 23:25:36.525057 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.524803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" event={"ID":"b9e66f44-da70-4361-a106-a2a731870f47","Type":"ContainerDied","Data":"2422201a987441049f8a459a888b41193f88ca7653f58a0a2a16071f67386366"} Sep 17 23:25:36.528353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.528270 2559 generic.go:356] "Generic (PLEG): container finished" podID="7be52dd4-470a-437c-9ad9-a749a110efe6" containerID="72cfc9c38283e479650595986f226ca84f9dbf793b8da467d72c98ebee180055" exitCode=0 Sep 17 23:25:36.528469 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.528427 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7850-b3181f" podUID="d25533bf-a2af-4951-8a5b-a932ae379a39" containerName="test-curl-7850-b3181f" containerID="cri-o://bd0ccaac46ddb473847cd3b7272a3dabdd1640c928aa34b45c93162fbac8b950" gracePeriod=2 Sep 17 23:25:36.528530 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.528468 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" event={"ID":"7be52dd4-470a-437c-9ad9-a749a110efe6","Type":"ContainerDied","Data":"72cfc9c38283e479650595986f226ca84f9dbf793b8da467d72c98ebee180055"} Sep 17 23:25:36.579552 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.579522 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762"] Sep 17 23:25:36.582139 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:25:36.582107 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1768fa61_e9d1_4147_bddc_1a1c283335aa.slice/crio-b70b7e9600900454058c71742e6beb5240329ca6de9e0235547e0bde57c47d73 WatchSource:0}: Error finding container b70b7e9600900454058c71742e6beb5240329ca6de9e0235547e0bde57c47d73: Status 404 returned error can't find the container with id b70b7e9600900454058c71742e6beb5240329ca6de9e0235547e0bde57c47d73 Sep 17 23:25:36.584717 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.584686 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:25:36.584860 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.584845 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:25:36.584996 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.584975 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:25:36.614109 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.614087 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" Sep 17 23:25:36.651908 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.651875 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" Sep 17 23:25:36.690264 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.690239 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-57ba95" Sep 17 23:25:36.752085 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.752050 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fk2v\" (UniqueName: \"kubernetes.io/projected/7be52dd4-470a-437c-9ad9-a749a110efe6-kube-api-access-8fk2v\") pod \"7be52dd4-470a-437c-9ad9-a749a110efe6\" (UID: \"7be52dd4-470a-437c-9ad9-a749a110efe6\") " Sep 17 23:25:36.752303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.752288 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rtfl\" (UniqueName: \"kubernetes.io/projected/b9e66f44-da70-4361-a106-a2a731870f47-kube-api-access-8rtfl\") pod \"b9e66f44-da70-4361-a106-a2a731870f47\" (UID: \"b9e66f44-da70-4361-a106-a2a731870f47\") " Sep 17 23:25:36.752422 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.752410 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be52dd4-470a-437c-9ad9-a749a110efe6-plugins-config-volume\") pod \"7be52dd4-470a-437c-9ad9-a749a110efe6\" (UID: \"7be52dd4-470a-437c-9ad9-a749a110efe6\") " Sep 17 23:25:36.752537 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.752527 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e66f44-da70-4361-a106-a2a731870f47-plugins-config-volume\") pod \"b9e66f44-da70-4361-a106-a2a731870f47\" (UID: \"b9e66f44-da70-4361-a106-a2a731870f47\") " Sep 17 23:25:36.752971 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.752939 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be52dd4-470a-437c-9ad9-a749a110efe6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7be52dd4-470a-437c-9ad9-a749a110efe6" (UID: "7be52dd4-470a-437c-9ad9-a749a110efe6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:25:36.753096 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.752982 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e66f44-da70-4361-a106-a2a731870f47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9e66f44-da70-4361-a106-a2a731870f47" (UID: "b9e66f44-da70-4361-a106-a2a731870f47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:25:36.754326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.754305 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be52dd4-470a-437c-9ad9-a749a110efe6-kube-api-access-8fk2v" (OuterVolumeSpecName: "kube-api-access-8fk2v") pod "7be52dd4-470a-437c-9ad9-a749a110efe6" (UID: "7be52dd4-470a-437c-9ad9-a749a110efe6"). InnerVolumeSpecName "kube-api-access-8fk2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:36.755182 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.755166 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e66f44-da70-4361-a106-a2a731870f47-kube-api-access-8rtfl" (OuterVolumeSpecName: "kube-api-access-8rtfl") pod "b9e66f44-da70-4361-a106-a2a731870f47" (UID: "b9e66f44-da70-4361-a106-a2a731870f47"). InnerVolumeSpecName "kube-api-access-8rtfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:36.854220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.854145 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftn74\" (UniqueName: \"kubernetes.io/projected/bbeae5da-78fc-4ae3-9824-97f4ad4b367a-kube-api-access-ftn74\") pod \"bbeae5da-78fc-4ae3-9824-97f4ad4b367a\" (UID: \"bbeae5da-78fc-4ae3-9824-97f4ad4b367a\") " Sep 17 23:25:36.854580 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.854554 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fk2v\" (UniqueName: \"kubernetes.io/projected/7be52dd4-470a-437c-9ad9-a749a110efe6-kube-api-access-8fk2v\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:25:36.854580 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.854575 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rtfl\" (UniqueName: \"kubernetes.io/projected/b9e66f44-da70-4361-a106-a2a731870f47-kube-api-access-8rtfl\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:25:36.854689 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.854584 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be52dd4-470a-437c-9ad9-a749a110efe6-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:25:36.854689 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.854594 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e66f44-da70-4361-a106-a2a731870f47-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:25:36.856644 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.856619 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbeae5da-78fc-4ae3-9824-97f4ad4b367a-kube-api-access-ftn74" (OuterVolumeSpecName: "kube-api-access-ftn74") pod "bbeae5da-78fc-4ae3-9824-97f4ad4b367a" (UID: "bbeae5da-78fc-4ae3-9824-97f4ad4b367a"). InnerVolumeSpecName "kube-api-access-ftn74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:36.955461 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:36.955424 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftn74\" (UniqueName: \"kubernetes.io/projected/bbeae5da-78fc-4ae3-9824-97f4ad4b367a-kube-api-access-ftn74\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:25:37.535004 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.534917 2559 generic.go:356] "Generic (PLEG): container finished" podID="bbeae5da-78fc-4ae3-9824-97f4ad4b367a" containerID="70226c75a9de2cb85bafca12059c4ac35ea2c03bae38449bf11786a7c8025779" exitCode=137 Sep 17 23:25:37.535004 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.534967 2559 scope.go:117] "RemoveContainer" containerID="70226c75a9de2cb85bafca12059c4ac35ea2c03bae38449bf11786a7c8025779" Sep 17 23:25:37.535468 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.534967 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-57ba95" Sep 17 23:25:37.536599 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.536565 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" Sep 17 23:25:37.536709 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.536614 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j" event={"ID":"b9e66f44-da70-4361-a106-a2a731870f47","Type":"ContainerDied","Data":"e1c39c020737a45bdb0d82d33771d97736c845b4de64e5c6ed3cad76987255f8"} Sep 17 23:25:37.538320 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.538295 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" event={"ID":"1768fa61-e9d1-4147-bddc-1a1c283335aa","Type":"ContainerStarted","Data":"90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f"} Sep 17 23:25:37.538417 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.538342 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" event={"ID":"1768fa61-e9d1-4147-bddc-1a1c283335aa","Type":"ContainerStarted","Data":"b70b7e9600900454058c71742e6beb5240329ca6de9e0235547e0bde57c47d73"} Sep 17 23:25:37.539735 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.539714 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" Sep 17 23:25:37.539829 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.539772 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx" event={"ID":"7be52dd4-470a-437c-9ad9-a749a110efe6","Type":"ContainerDied","Data":"58a12cabf67d8d06bab46e96c7dcbf3b1e29a71b492a10671701f19760751e99"} Sep 17 23:25:37.547300 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.547241 2559 scope.go:117] "RemoveContainer" containerID="70226c75a9de2cb85bafca12059c4ac35ea2c03bae38449bf11786a7c8025779" Sep 17 23:25:37.547581 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:37.547557 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70226c75a9de2cb85bafca12059c4ac35ea2c03bae38449bf11786a7c8025779\": container with ID starting with 70226c75a9de2cb85bafca12059c4ac35ea2c03bae38449bf11786a7c8025779 not found: ID does not exist" containerID="70226c75a9de2cb85bafca12059c4ac35ea2c03bae38449bf11786a7c8025779" Sep 17 23:25:37.547677 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.547589 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70226c75a9de2cb85bafca12059c4ac35ea2c03bae38449bf11786a7c8025779"} err="failed to get container status \"70226c75a9de2cb85bafca12059c4ac35ea2c03bae38449bf11786a7c8025779\": rpc error: code = NotFound desc = could not find container \"70226c75a9de2cb85bafca12059c4ac35ea2c03bae38449bf11786a7c8025779\": container with ID starting with 70226c75a9de2cb85bafca12059c4ac35ea2c03bae38449bf11786a7c8025779 not found: ID does not exist" Sep 17 23:25:37.547677 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.547613 2559 scope.go:117] "RemoveContainer" containerID="2422201a987441049f8a459a888b41193f88ca7653f58a0a2a16071f67386366" Sep 17 23:25:37.556496 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.556478 2559 scope.go:117] "RemoveContainer" containerID="72cfc9c38283e479650595986f226ca84f9dbf793b8da467d72c98ebee180055" Sep 17 23:25:37.565422 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.565372 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" podStartSLOduration=2.565355277 podStartE2EDuration="2.565355277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:37.561966924 +0000 UTC m=+1165.679446395" watchObservedRunningTime="2026-09-17 23:25:37.565355277 +0000 UTC m=+1165.682834727" Sep 17 23:25:37.575339 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.575319 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j"] Sep 17 23:25:37.578169 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:37.578143 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7850-57ba95\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="bbeae5da-78fc-4ae3-9824-97f4ad4b367a" pod="opendatahub/test-curl-7850-57ba95" Sep 17 23:25:37.579011 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.578993 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48d98f-6bcb4f8579-xj62j"] Sep 17 23:25:37.590488 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.590467 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx"] Sep 17 23:25:37.600674 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:37.595965 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48d98f-86f7687cf8-8dbhx"] Sep 17 23:25:38.280126 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:38.280089 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:38.288939 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:38.288889 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7be52dd4-470a-437c-9ad9-a749a110efe6" path="/var/lib/kubelet/pods/7be52dd4-470a-437c-9ad9-a749a110efe6/volumes" Sep 17 23:25:38.289495 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:38.289477 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e66f44-da70-4361-a106-a2a731870f47" path="/var/lib/kubelet/pods/b9e66f44-da70-4361-a106-a2a731870f47/volumes" Sep 17 23:25:38.289913 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:38.289887 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbeae5da-78fc-4ae3-9824-97f4ad4b367a" path="/var/lib/kubelet/pods/bbeae5da-78fc-4ae3-9824-97f4ad4b367a/volumes" Sep 17 23:25:38.290316 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:38.290302 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:38.550797 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:38.550771 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:38.551775 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:38.551755 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:25:38.722776 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:38.722751 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-b3181f" Sep 17 23:25:38.772883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:38.772855 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-677w4\" (UniqueName: \"kubernetes.io/projected/d25533bf-a2af-4951-8a5b-a932ae379a39-kube-api-access-677w4\") pod \"d25533bf-a2af-4951-8a5b-a932ae379a39\" (UID: \"d25533bf-a2af-4951-8a5b-a932ae379a39\") " Sep 17 23:25:38.774997 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:38.774971 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25533bf-a2af-4951-8a5b-a932ae379a39-kube-api-access-677w4" (OuterVolumeSpecName: "kube-api-access-677w4") pod "d25533bf-a2af-4951-8a5b-a932ae379a39" (UID: "d25533bf-a2af-4951-8a5b-a932ae379a39"). InnerVolumeSpecName "kube-api-access-677w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:38.874354 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:38.874291 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-677w4\" (UniqueName: \"kubernetes.io/projected/d25533bf-a2af-4951-8a5b-a932ae379a39-kube-api-access-677w4\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:25:39.557844 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:39.557807 2559 generic.go:356] "Generic (PLEG): container finished" podID="d25533bf-a2af-4951-8a5b-a932ae379a39" containerID="bd0ccaac46ddb473847cd3b7272a3dabdd1640c928aa34b45c93162fbac8b950" exitCode=137 Sep 17 23:25:39.558259 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:39.557859 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7850-b3181f" Sep 17 23:25:39.558259 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:39.557931 2559 scope.go:117] "RemoveContainer" containerID="bd0ccaac46ddb473847cd3b7272a3dabdd1640c928aa34b45c93162fbac8b950" Sep 17 23:25:39.567018 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:39.567000 2559 scope.go:117] "RemoveContainer" containerID="bd0ccaac46ddb473847cd3b7272a3dabdd1640c928aa34b45c93162fbac8b950" Sep 17 23:25:39.567262 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:39.567244 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0ccaac46ddb473847cd3b7272a3dabdd1640c928aa34b45c93162fbac8b950\": container with ID starting with bd0ccaac46ddb473847cd3b7272a3dabdd1640c928aa34b45c93162fbac8b950 not found: ID does not exist" containerID="bd0ccaac46ddb473847cd3b7272a3dabdd1640c928aa34b45c93162fbac8b950" Sep 17 23:25:39.567309 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:39.567270 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0ccaac46ddb473847cd3b7272a3dabdd1640c928aa34b45c93162fbac8b950"} err="failed to get container status \"bd0ccaac46ddb473847cd3b7272a3dabdd1640c928aa34b45c93162fbac8b950\": rpc error: code = NotFound desc = could not find container \"bd0ccaac46ddb473847cd3b7272a3dabdd1640c928aa34b45c93162fbac8b950\": container with ID starting with bd0ccaac46ddb473847cd3b7272a3dabdd1640c928aa34b45c93162fbac8b950 not found: ID does not exist" Sep 17 23:25:39.568480 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:25:39.568454 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7850-b3181f\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="d25533bf-a2af-4951-8a5b-a932ae379a39" pod="opendatahub/test-curl-7850-b3181f" Sep 17 23:25:40.288512 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:40.288480 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d25533bf-a2af-4951-8a5b-a932ae379a39" path="/var/lib/kubelet/pods/d25533bf-a2af-4951-8a5b-a932ae379a39/volumes" Sep 17 23:25:42.479039 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.479002 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p"] Sep 17 23:25:42.479594 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.479576 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7be52dd4-470a-437c-9ad9-a749a110efe6" containerName="payload-pre-processing" Sep 17 23:25:42.479594 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.479594 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7be52dd4-470a-437c-9ad9-a749a110efe6" containerName="payload-pre-processing" Sep 17 23:25:42.479704 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.479619 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e66f44-da70-4361-a106-a2a731870f47" containerName="payload-processing" Sep 17 23:25:42.479704 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.479625 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e66f44-da70-4361-a106-a2a731870f47" containerName="payload-processing" Sep 17 23:25:42.479704 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.479696 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7be52dd4-470a-437c-9ad9-a749a110efe6" containerName="payload-pre-processing" Sep 17 23:25:42.479704 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.479703 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e66f44-da70-4361-a106-a2a731870f47" containerName="payload-processing" Sep 17 23:25:42.484286 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.484269 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" Sep 17 23:25:42.486755 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.486732 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-09450b8d-dockercfg-8dqmt\"" Sep 17 23:25:42.486849 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.486732 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-09450b8d\"" Sep 17 23:25:42.493638 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.493615 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p"] Sep 17 23:25:42.509370 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.509339 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z726\" (UniqueName: \"kubernetes.io/projected/097695a0-bcfd-4dc4-a37c-8b1907d9899f-kube-api-access-6z726\") pod \"payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p\" (UID: \"097695a0-bcfd-4dc4-a37c-8b1907d9899f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" Sep 17 23:25:42.509485 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.509383 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097695a0-bcfd-4dc4-a37c-8b1907d9899f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p\" (UID: \"097695a0-bcfd-4dc4-a37c-8b1907d9899f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" Sep 17 23:25:42.610894 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.610860 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z726\" (UniqueName: \"kubernetes.io/projected/097695a0-bcfd-4dc4-a37c-8b1907d9899f-kube-api-access-6z726\") pod \"payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p\" (UID: \"097695a0-bcfd-4dc4-a37c-8b1907d9899f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" Sep 17 23:25:42.611061 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.610938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097695a0-bcfd-4dc4-a37c-8b1907d9899f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p\" (UID: \"097695a0-bcfd-4dc4-a37c-8b1907d9899f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" Sep 17 23:25:42.611643 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.611626 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097695a0-bcfd-4dc4-a37c-8b1907d9899f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p\" (UID: \"097695a0-bcfd-4dc4-a37c-8b1907d9899f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" Sep 17 23:25:42.619238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.619212 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z726\" (UniqueName: \"kubernetes.io/projected/097695a0-bcfd-4dc4-a37c-8b1907d9899f-kube-api-access-6z726\") pod \"payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p\" (UID: \"097695a0-bcfd-4dc4-a37c-8b1907d9899f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" Sep 17 23:25:42.795961 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:42.795874 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" Sep 17 23:25:43.069711 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:43.069687 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p"] Sep 17 23:25:43.071050 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:25:43.071023 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097695a0_bcfd_4dc4_a37c_8b1907d9899f.slice/crio-7239680bd02a53c041620af0faf077d7f03d435965877a00afc6b5357598abce WatchSource:0}: Error finding container 7239680bd02a53c041620af0faf077d7f03d435965877a00afc6b5357598abce: Status 404 returned error can't find the container with id 7239680bd02a53c041620af0faf077d7f03d435965877a00afc6b5357598abce Sep 17 23:25:43.585574 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:43.585537 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" event={"ID":"097695a0-bcfd-4dc4-a37c-8b1907d9899f","Type":"ContainerStarted","Data":"835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb"} Sep 17 23:25:43.586050 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:43.585594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" event={"ID":"097695a0-bcfd-4dc4-a37c-8b1907d9899f","Type":"ContainerStarted","Data":"7239680bd02a53c041620af0faf077d7f03d435965877a00afc6b5357598abce"} Sep 17 23:25:43.586050 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:43.585630 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" Sep 17 23:25:43.604356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:43.604308 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" podStartSLOduration=1.604290551 podStartE2EDuration="1.604290551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:43.602385248 +0000 UTC m=+1171.719864722" watchObservedRunningTime="2026-09-17 23:25:43.604290551 +0000 UTC m=+1171.721769999" Sep 17 23:25:54.594498 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:25:54.594469 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" Sep 17 23:26:03.100643 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:03.100596 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:26:13.764433 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:13.764396 2559 generic.go:356] "Generic (PLEG): container finished" podID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerID="90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f" exitCode=0 Sep 17 23:26:13.764858 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:13.764449 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" event={"ID":"1768fa61-e9d1-4147-bddc-1a1c283335aa","Type":"ContainerDied","Data":"90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f"} Sep 17 23:26:13.764858 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:13.764789 2559 scope.go:117] "RemoveContainer" containerID="90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f" Sep 17 23:26:13.766018 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:13.765989 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:26:13.766086 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:13.766057 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:26:13.766150 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:13.766087 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:26:14.261129 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:14.261102 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:26:14.264499 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:14.264479 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:26:14.771353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:14.771320 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" event={"ID":"1768fa61-e9d1-4147-bddc-1a1c283335aa","Type":"ContainerStarted","Data":"c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c"} Sep 17 23:26:15.280333 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:15.280238 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:26:15.280548 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:15.280524 2559 patch_prober.go:28] interesting pod/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 17 23:26:15.280617 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:15.280585 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 17 23:26:16.280808 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:16.280774 2559 patch_prober.go:28] interesting pod/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 17 23:26:16.281185 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:16.280827 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 17 23:26:17.280769 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:17.280724 2559 patch_prober.go:28] interesting pod/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 17 23:26:17.280991 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:17.280795 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 17 23:26:18.280206 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:18.280172 2559 patch_prober.go:28] interesting pod/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 17 23:26:18.280380 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:18.280220 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 17 23:26:19.281144 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:19.281107 2559 patch_prober.go:28] interesting pod/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 17 23:26:19.281528 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:19.281161 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 17 23:26:20.280296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:20.280266 2559 patch_prober.go:28] interesting pod/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 17 23:26:20.280467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:20.280319 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 17 23:26:21.285414 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:21.285381 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:26:21.285886 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:21.285859 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:26:21.286879 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:21.286858 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:26:30.067054 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.067015 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv"] Sep 17 23:26:30.067607 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.067230 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" podUID="f39bb2dc-7241-4137-aea9-5ef0c6b5a240" containerName="payload-processing" containerID="cri-o://7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1" gracePeriod=30 Sep 17 23:26:30.338786 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.338762 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" Sep 17 23:26:30.366921 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.366872 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4v4b\" (UniqueName: \"kubernetes.io/projected/f39bb2dc-7241-4137-aea9-5ef0c6b5a240-kube-api-access-f4v4b\") pod \"f39bb2dc-7241-4137-aea9-5ef0c6b5a240\" (UID: \"f39bb2dc-7241-4137-aea9-5ef0c6b5a240\") " Sep 17 23:26:30.367398 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.367374 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39bb2dc-7241-4137-aea9-5ef0c6b5a240-plugins-config-volume\") pod \"f39bb2dc-7241-4137-aea9-5ef0c6b5a240\" (UID: \"f39bb2dc-7241-4137-aea9-5ef0c6b5a240\") " Sep 17 23:26:30.367977 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.367868 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39bb2dc-7241-4137-aea9-5ef0c6b5a240-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f39bb2dc-7241-4137-aea9-5ef0c6b5a240" (UID: "f39bb2dc-7241-4137-aea9-5ef0c6b5a240"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:26:30.369565 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.369531 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39bb2dc-7241-4137-aea9-5ef0c6b5a240-kube-api-access-f4v4b" (OuterVolumeSpecName: "kube-api-access-f4v4b") pod "f39bb2dc-7241-4137-aea9-5ef0c6b5a240" (UID: "f39bb2dc-7241-4137-aea9-5ef0c6b5a240"). InnerVolumeSpecName "kube-api-access-f4v4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:26:30.469443 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.469408 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39bb2dc-7241-4137-aea9-5ef0c6b5a240-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:26:30.469443 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.469434 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4v4b\" (UniqueName: \"kubernetes.io/projected/f39bb2dc-7241-4137-aea9-5ef0c6b5a240-kube-api-access-f4v4b\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:26:30.861781 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.861741 2559 generic.go:356] "Generic (PLEG): container finished" podID="f39bb2dc-7241-4137-aea9-5ef0c6b5a240" containerID="7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1" exitCode=0 Sep 17 23:26:30.861781 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.861785 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" Sep 17 23:26:30.862001 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.861835 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" event={"ID":"f39bb2dc-7241-4137-aea9-5ef0c6b5a240","Type":"ContainerDied","Data":"7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1"} Sep 17 23:26:30.862001 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.861885 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv" event={"ID":"f39bb2dc-7241-4137-aea9-5ef0c6b5a240","Type":"ContainerDied","Data":"a1f3cbeec30c40dc131613b6084bb2213150dab1a6319093f0a228aa37da04a9"} Sep 17 23:26:30.862001 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.861937 2559 scope.go:117] "RemoveContainer" containerID="7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1" Sep 17 23:26:30.871218 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.871202 2559 scope.go:117] "RemoveContainer" containerID="7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1" Sep 17 23:26:30.871492 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:26:30.871474 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1\": container with ID starting with 7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1 not found: ID does not exist" containerID="7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1" Sep 17 23:26:30.871543 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.871499 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1"} err="failed to get container status \"7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1\": rpc error: code = NotFound desc = could not find container \"7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1\": container with ID starting with 7a61fda96cdfd6ae766a4ec65201d37d79cdb5036aad17d400309aaa206c65c1 not found: ID does not exist" Sep 17 23:26:30.882451 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.882426 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv"] Sep 17 23:26:30.888129 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:30.888109 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f38e68-5d4f46496c-pgvsv"] Sep 17 23:26:32.289786 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:32.289749 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f39bb2dc-7241-4137-aea9-5ef0c6b5a240" path="/var/lib/kubelet/pods/f39bb2dc-7241-4137-aea9-5ef0c6b5a240/volumes" Sep 17 23:26:42.439545 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.439503 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t"] Sep 17 23:26:42.440077 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.439961 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39bb2dc-7241-4137-aea9-5ef0c6b5a240" containerName="payload-processing" Sep 17 23:26:42.440077 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.439975 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39bb2dc-7241-4137-aea9-5ef0c6b5a240" containerName="payload-processing" Sep 17 23:26:42.440220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.440090 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f39bb2dc-7241-4137-aea9-5ef0c6b5a240" containerName="payload-processing" Sep 17 23:26:42.444814 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.444790 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.447798 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.447764 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-f80c49-openshift-default-dockercfg-tdsw9\"" Sep 17 23:26:42.455886 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.455866 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t"] Sep 17 23:26:42.477984 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.477952 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-podinfo\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.478116 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.477997 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g255\" (UniqueName: \"kubernetes.io/projected/d7c41495-4ea9-4c22-a74d-b273f369cc31-kube-api-access-6g255\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.478116 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.478019 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-token\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.478116 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.478099 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c41495-4ea9-4c22-a74d-b273f369cc31-istiod-ca-cert\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.478278 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.478132 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-workload-socket\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.478278 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.478151 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-data\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.478278 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.478168 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-envoy\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.478422 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.478295 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-workload-certs\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.478422 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.478342 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-credential-socket\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579123 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-workload-socket\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579313 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579153 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-data\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579313 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579184 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-envoy\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579313 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-workload-certs\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579313 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579269 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-credential-socket\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579548 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579388 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-workload-socket\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579548 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579425 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-podinfo\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579548 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579462 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g255\" (UniqueName: \"kubernetes.io/projected/d7c41495-4ea9-4c22-a74d-b273f369cc31-kube-api-access-6g255\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579548 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-token\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-data\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-credential-socket\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579551 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c41495-4ea9-4c22-a74d-b273f369cc31-istiod-ca-cert\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.579753 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.579643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-workload-certs\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.580182 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.580161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c41495-4ea9-4c22-a74d-b273f369cc31-istiod-ca-cert\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.581532 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.581503 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-envoy\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.581803 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.581783 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-podinfo\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.587545 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.587523 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-token\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.587687 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.587671 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g255\" (UniqueName: \"kubernetes.io/projected/d7c41495-4ea9-4c22-a74d-b273f369cc31-kube-api-access-6g255\") pod \"e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:42.758550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:42.758519 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:43.032626 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:43.032585 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t"] Sep 17 23:26:43.033658 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:26:43.033632 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c41495_4ea9_4c22_a74d_b273f369cc31.slice/crio-9798383f9d6278a0902be320606c75ad0f809542b2151cd76e88979fa121cb64 WatchSource:0}: Error finding container 9798383f9d6278a0902be320606c75ad0f809542b2151cd76e88979fa121cb64: Status 404 returned error can't find the container with id 9798383f9d6278a0902be320606c75ad0f809542b2151cd76e88979fa121cb64 Sep 17 23:26:43.036029 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:43.035992 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:26:43.036163 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:43.036061 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:26:43.036163 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:43.036091 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:26:43.932376 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:43.932340 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" event={"ID":"d7c41495-4ea9-4c22-a74d-b273f369cc31","Type":"ContainerStarted","Data":"dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335"} Sep 17 23:26:43.932840 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:43.932431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" event={"ID":"d7c41495-4ea9-4c22-a74d-b273f369cc31","Type":"ContainerStarted","Data":"9798383f9d6278a0902be320606c75ad0f809542b2151cd76e88979fa121cb64"} Sep 17 23:26:43.954651 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:43.954607 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" podStartSLOduration=1.954593108 podStartE2EDuration="1.954593108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:26:43.952500142 +0000 UTC m=+1232.069979602" watchObservedRunningTime="2026-09-17 23:26:43.954593108 +0000 UTC m=+1232.072072557" Sep 17 23:26:44.758859 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:44.758823 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:44.763995 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:44.763970 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:44.938670 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:44.938642 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:44.939650 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:44.939634 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:26:45.926416 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:45.926385 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:26:54.568772 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:54.568737 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d"] Sep 17 23:26:54.569264 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:54.568946 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" podUID="6009324f-0107-4cd3-9890-9fa7521de183" containerName="istio-proxy" containerID="cri-o://745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74" gracePeriod=30 Sep 17 23:26:56.130347 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:56.130260 2559 patch_prober.go:28] interesting pod/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 23:26:56.130347 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:56.130328 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" podUID="6009324f-0107-4cd3-9890-9fa7521de183" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 23:26:59.817336 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.817313 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:26:59.949944 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.949885 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6009324f-0107-4cd3-9890-9fa7521de183-istio-podinfo\") pod \"6009324f-0107-4cd3-9890-9fa7521de183\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " Sep 17 23:26:59.950096 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950004 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-credential-socket\") pod \"6009324f-0107-4cd3-9890-9fa7521de183\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " Sep 17 23:26:59.950096 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950042 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6009324f-0107-4cd3-9890-9fa7521de183-istio-token\") pod \"6009324f-0107-4cd3-9890-9fa7521de183\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " Sep 17 23:26:59.950096 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950072 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-istio-envoy\") pod \"6009324f-0107-4cd3-9890-9fa7521de183\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " Sep 17 23:26:59.950220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950155 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6009324f-0107-4cd3-9890-9fa7521de183-istiod-ca-cert\") pod \"6009324f-0107-4cd3-9890-9fa7521de183\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " Sep 17 23:26:59.950220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950199 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-workload-certs\") pod \"6009324f-0107-4cd3-9890-9fa7521de183\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " Sep 17 23:26:59.950321 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950249 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-istio-data\") pod \"6009324f-0107-4cd3-9890-9fa7521de183\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " Sep 17 23:26:59.950321 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950280 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-workload-socket\") pod \"6009324f-0107-4cd3-9890-9fa7521de183\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " Sep 17 23:26:59.950321 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950281 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6009324f-0107-4cd3-9890-9fa7521de183" (UID: "6009324f-0107-4cd3-9890-9fa7521de183"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:26:59.950321 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950309 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbhbx\" (UniqueName: \"kubernetes.io/projected/6009324f-0107-4cd3-9890-9fa7521de183-kube-api-access-zbhbx\") pod \"6009324f-0107-4cd3-9890-9fa7521de183\" (UID: \"6009324f-0107-4cd3-9890-9fa7521de183\") " Sep 17 23:26:59.950523 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950456 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-istio-data" (OuterVolumeSpecName: "istio-data") pod "6009324f-0107-4cd3-9890-9fa7521de183" (UID: "6009324f-0107-4cd3-9890-9fa7521de183"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:26:59.950582 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950521 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6009324f-0107-4cd3-9890-9fa7521de183-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6009324f-0107-4cd3-9890-9fa7521de183" (UID: "6009324f-0107-4cd3-9890-9fa7521de183"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:26:59.950752 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950686 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:26:59.950752 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950696 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6009324f-0107-4cd3-9890-9fa7521de183" (UID: "6009324f-0107-4cd3-9890-9fa7521de183"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:26:59.950752 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950710 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6009324f-0107-4cd3-9890-9fa7521de183-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:26:59.950752 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950725 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:26:59.951106 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.950865 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6009324f-0107-4cd3-9890-9fa7521de183" (UID: "6009324f-0107-4cd3-9890-9fa7521de183"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:26:59.952264 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.952239 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6009324f-0107-4cd3-9890-9fa7521de183-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6009324f-0107-4cd3-9890-9fa7521de183" (UID: "6009324f-0107-4cd3-9890-9fa7521de183"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:26:59.952566 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.952538 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6009324f-0107-4cd3-9890-9fa7521de183" (UID: "6009324f-0107-4cd3-9890-9fa7521de183"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:26:59.952638 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.952543 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6009324f-0107-4cd3-9890-9fa7521de183-istio-token" (OuterVolumeSpecName: "istio-token") pod "6009324f-0107-4cd3-9890-9fa7521de183" (UID: "6009324f-0107-4cd3-9890-9fa7521de183"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:26:59.952833 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:26:59.952814 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6009324f-0107-4cd3-9890-9fa7521de183-kube-api-access-zbhbx" (OuterVolumeSpecName: "kube-api-access-zbhbx") pod "6009324f-0107-4cd3-9890-9fa7521de183" (UID: "6009324f-0107-4cd3-9890-9fa7521de183"). InnerVolumeSpecName "kube-api-access-zbhbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:27:00.022369 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.022345 2559 generic.go:356] "Generic (PLEG): container finished" podID="6009324f-0107-4cd3-9890-9fa7521de183" containerID="745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74" exitCode=0 Sep 17 23:27:00.022464 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.022396 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" Sep 17 23:27:00.022464 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.022428 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" event={"ID":"6009324f-0107-4cd3-9890-9fa7521de183","Type":"ContainerDied","Data":"745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74"} Sep 17 23:27:00.022538 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.022502 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d" event={"ID":"6009324f-0107-4cd3-9890-9fa7521de183","Type":"ContainerDied","Data":"9190b8d16c6e5c9f9eff1796e1af8347bef455bdf533fce4c7ff0d08c20cac16"} Sep 17 23:27:00.022538 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.022524 2559 scope.go:117] "RemoveContainer" containerID="745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74" Sep 17 23:27:00.031716 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.031700 2559 scope.go:117] "RemoveContainer" containerID="745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74" Sep 17 23:27:00.032024 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:27:00.032005 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74\": container with ID starting with 745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74 not found: ID does not exist" containerID="745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74" Sep 17 23:27:00.032081 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.032029 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74"} err="failed to get container status \"745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74\": rpc error: code = NotFound desc = could not find container \"745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74\": container with ID starting with 745c50c9ae1df3f897302e500bbb60b4402f776654b3131aab7b617c73bd0e74 not found: ID does not exist" Sep 17 23:27:00.046480 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.046459 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d"] Sep 17 23:27:00.049764 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.049746 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-be3eb4-openshift-default-684dc4b59d-qx67d"] Sep 17 23:27:00.051954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.051935 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:27:00.052053 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.051957 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:27:00.052053 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.051972 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbhbx\" (UniqueName: \"kubernetes.io/projected/6009324f-0107-4cd3-9890-9fa7521de183-kube-api-access-zbhbx\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:27:00.052053 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.051985 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6009324f-0107-4cd3-9890-9fa7521de183-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:27:00.052053 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.051997 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6009324f-0107-4cd3-9890-9fa7521de183-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:27:00.052053 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.052008 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6009324f-0107-4cd3-9890-9fa7521de183-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:27:00.287853 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:00.287763 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6009324f-0107-4cd3-9890-9fa7521de183" path="/var/lib/kubelet/pods/6009324f-0107-4cd3-9890-9fa7521de183/volumes" Sep 17 23:27:13.105629 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:13.105595 2559 generic.go:356] "Generic (PLEG): container finished" podID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerID="dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335" exitCode=0 Sep 17 23:27:13.106067 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:13.105666 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" event={"ID":"d7c41495-4ea9-4c22-a74d-b273f369cc31","Type":"ContainerDied","Data":"dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335"} Sep 17 23:27:13.106067 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:13.106061 2559 scope.go:117] "RemoveContainer" containerID="dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335" Sep 17 23:27:13.107231 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:13.107203 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:27:13.107292 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:13.107271 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:27:13.107328 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:13.107303 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:27:14.113530 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:14.113497 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" event={"ID":"d7c41495-4ea9-4c22-a74d-b273f369cc31","Type":"ContainerStarted","Data":"a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735"} Sep 17 23:27:14.759874 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:14.759833 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:27:14.761169 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:14.761135 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 17 23:27:14.761289 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:14.761198 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 17 23:27:14.938939 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:14.938893 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:27:15.030220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:15.030140 2559 scope.go:117] "RemoveContainer" containerID="2ba9cec8b0d888cb30d6b38a19a844957e69f329b26b8ad3cca4b42b89be0e82" Sep 17 23:27:15.777381 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:15.777348 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 17 23:27:15.778340 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:15.778309 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 17 23:27:16.759789 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:16.759749 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 17 23:27:16.760136 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:16.760100 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 17 23:27:17.468762 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.468719 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85"] Sep 17 23:27:17.469949 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.469913 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6009324f-0107-4cd3-9890-9fa7521de183" containerName="istio-proxy" Sep 17 23:27:17.470100 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.470088 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6009324f-0107-4cd3-9890-9fa7521de183" containerName="istio-proxy" Sep 17 23:27:17.470338 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.470324 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6009324f-0107-4cd3-9890-9fa7521de183" containerName="istio-proxy" Sep 17 23:27:17.473708 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.473692 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:17.476556 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.476525 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-f6ac6e-serving-cert\"" Sep 17 23:27:17.487931 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.486205 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85"] Sep 17 23:27:17.516199 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.516173 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd59b83-c48d-4c53-9fdd-93850998ef76-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:17.516394 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.516371 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:17.516533 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.516516 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsgfv\" (UniqueName: \"kubernetes.io/projected/8cd59b83-c48d-4c53-9fdd-93850998ef76-kube-api-access-wsgfv\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:17.516603 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.516582 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:17.617353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.617315 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsgfv\" (UniqueName: \"kubernetes.io/projected/8cd59b83-c48d-4c53-9fdd-93850998ef76-kube-api-access-wsgfv\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:17.617613 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.617596 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:17.617862 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.617826 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd59b83-c48d-4c53-9fdd-93850998ef76-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:17.618000 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.617986 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:17.618122 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:27:17.618105 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-serving-cert: secret "maas-api-e2e-shared-b-w3-f6ac6e-serving-cert" not found Sep 17 23:27:17.618187 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:27:17.618178 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-maas-api-tls podName:8cd59b83-c48d-4c53-9fdd-93850998ef76 nodeName:}" failed. No retries permitted until 2026-09-17 23:27:18.118162573 +0000 UTC m=+1266.235642005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-maas-api-tls") pod "maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" (UID: "8cd59b83-c48d-4c53-9fdd-93850998ef76") : secret "maas-api-e2e-shared-b-w3-f6ac6e-serving-cert" not found Sep 17 23:27:17.618439 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.618421 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd59b83-c48d-4c53-9fdd-93850998ef76-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:17.619893 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.619877 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:17.628672 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.628652 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsgfv\" (UniqueName: \"kubernetes.io/projected/8cd59b83-c48d-4c53-9fdd-93850998ef76-kube-api-access-wsgfv\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:17.725158 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.725081 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:27:17.759625 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.759197 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 17 23:27:17.759625 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:17.759260 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 17 23:27:18.124878 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.124792 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:18.127918 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.127875 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:18.129480 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.129456 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq"] Sep 17 23:27:18.132427 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.132406 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" Sep 17 23:27:18.135131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.135107 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-f6ac6e-dockercfg-8989c\"" Sep 17 23:27:18.135373 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.135356 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f6ac6e\"" Sep 17 23:27:18.155929 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.151867 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq"] Sep 17 23:27:18.226068 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.226027 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2842d6c8-adbb-4231-93f4-002f19cd2dd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq\" (UID: \"2842d6c8-adbb-4231-93f4-002f19cd2dd0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" Sep 17 23:27:18.226348 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.226327 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9thc\" (UniqueName: \"kubernetes.io/projected/2842d6c8-adbb-4231-93f4-002f19cd2dd0-kube-api-access-v9thc\") pod \"payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq\" (UID: \"2842d6c8-adbb-4231-93f4-002f19cd2dd0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" Sep 17 23:27:18.327321 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.327289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9thc\" (UniqueName: \"kubernetes.io/projected/2842d6c8-adbb-4231-93f4-002f19cd2dd0-kube-api-access-v9thc\") pod \"payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq\" (UID: \"2842d6c8-adbb-4231-93f4-002f19cd2dd0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" Sep 17 23:27:18.327523 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.327420 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2842d6c8-adbb-4231-93f4-002f19cd2dd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq\" (UID: \"2842d6c8-adbb-4231-93f4-002f19cd2dd0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" Sep 17 23:27:18.328090 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.328069 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2842d6c8-adbb-4231-93f4-002f19cd2dd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq\" (UID: \"2842d6c8-adbb-4231-93f4-002f19cd2dd0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" Sep 17 23:27:18.336493 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.336471 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9thc\" (UniqueName: \"kubernetes.io/projected/2842d6c8-adbb-4231-93f4-002f19cd2dd0-kube-api-access-v9thc\") pod \"payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq\" (UID: \"2842d6c8-adbb-4231-93f4-002f19cd2dd0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" Sep 17 23:27:18.388556 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.388479 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:18.486667 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.479613 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" Sep 17 23:27:18.671016 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.670923 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85"] Sep 17 23:27:18.675112 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:27:18.675086 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd59b83_c48d_4c53_9fdd_93850998ef76.slice/crio-7eb87f09e11f22607cb28e78ae62e3164106fa29fab93456190d863e8fe82843 WatchSource:0}: Error finding container 7eb87f09e11f22607cb28e78ae62e3164106fa29fab93456190d863e8fe82843: Status 404 returned error can't find the container with id 7eb87f09e11f22607cb28e78ae62e3164106fa29fab93456190d863e8fe82843 Sep 17 23:27:18.759349 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.759306 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 17 23:27:18.759540 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.759375 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 17 23:27:18.803746 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:18.803719 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq"] Sep 17 23:27:18.805331 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:27:18.805306 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2842d6c8_adbb_4231_93f4_002f19cd2dd0.slice/crio-7b475b2d84351c43f91d735407fbee09a0d115af9c1ba19f7e2ec9012d8a5091 WatchSource:0}: Error finding container 7b475b2d84351c43f91d735407fbee09a0d115af9c1ba19f7e2ec9012d8a5091: Status 404 returned error can't find the container with id 7b475b2d84351c43f91d735407fbee09a0d115af9c1ba19f7e2ec9012d8a5091 Sep 17 23:27:19.146167 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:19.146123 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" event={"ID":"8cd59b83-c48d-4c53-9fdd-93850998ef76","Type":"ContainerStarted","Data":"6346e6375b69454a23f29cf95761d3fae230b11628511077e7df33e4c737653d"} Sep 17 23:27:19.146361 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:19.146195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" event={"ID":"8cd59b83-c48d-4c53-9fdd-93850998ef76","Type":"ContainerStarted","Data":"7eb87f09e11f22607cb28e78ae62e3164106fa29fab93456190d863e8fe82843"} Sep 17 23:27:19.147031 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:19.147005 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:19.148765 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:19.148734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" event={"ID":"2842d6c8-adbb-4231-93f4-002f19cd2dd0","Type":"ContainerStarted","Data":"4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b"} Sep 17 23:27:19.148868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:19.148786 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" event={"ID":"2842d6c8-adbb-4231-93f4-002f19cd2dd0","Type":"ContainerStarted","Data":"7b475b2d84351c43f91d735407fbee09a0d115af9c1ba19f7e2ec9012d8a5091"} Sep 17 23:27:19.149298 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:19.149277 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" Sep 17 23:27:19.167385 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:19.167328 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" podStartSLOduration=1.875745503 podStartE2EDuration="2.16730963s" totalImagesPullingTime="291.564127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:19.164211439 +0000 UTC m=+1267.281690917" watchObservedRunningTime="2026-09-17 23:27:19.16730963 +0000 UTC m=+1267.284789084" Sep 17 23:27:19.183719 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:19.183674 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" podStartSLOduration=1.183654446 podStartE2EDuration="1.183654446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:19.182354803 +0000 UTC m=+1267.299834254" watchObservedRunningTime="2026-09-17 23:27:19.183654446 +0000 UTC m=+1267.301133896" Sep 17 23:27:19.764094 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:19.764061 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:27:19.765211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:19.765191 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:27:26.166790 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:26.166761 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:27:31.163559 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:31.163524 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" Sep 17 23:27:42.708492 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:27:42.708459 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:28:17.582060 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:17.582025 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:28:17.695383 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:17.695347 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p"] Sep 17 23:28:17.695624 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:17.695577 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" podUID="097695a0-bcfd-4dc4-a37c-8b1907d9899f" containerName="payload-pre-processing" containerID="cri-o://835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb" gracePeriod=30 Sep 17 23:28:17.965498 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:17.965476 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" Sep 17 23:28:18.086389 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.086354 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097695a0-bcfd-4dc4-a37c-8b1907d9899f-plugins-config-volume\") pod \"097695a0-bcfd-4dc4-a37c-8b1907d9899f\" (UID: \"097695a0-bcfd-4dc4-a37c-8b1907d9899f\") " Sep 17 23:28:18.086567 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.086458 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z726\" (UniqueName: \"kubernetes.io/projected/097695a0-bcfd-4dc4-a37c-8b1907d9899f-kube-api-access-6z726\") pod \"097695a0-bcfd-4dc4-a37c-8b1907d9899f\" (UID: \"097695a0-bcfd-4dc4-a37c-8b1907d9899f\") " Sep 17 23:28:18.086799 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.086774 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097695a0-bcfd-4dc4-a37c-8b1907d9899f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "097695a0-bcfd-4dc4-a37c-8b1907d9899f" (UID: "097695a0-bcfd-4dc4-a37c-8b1907d9899f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:28:18.088525 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.088501 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097695a0-bcfd-4dc4-a37c-8b1907d9899f-kube-api-access-6z726" (OuterVolumeSpecName: "kube-api-access-6z726") pod "097695a0-bcfd-4dc4-a37c-8b1907d9899f" (UID: "097695a0-bcfd-4dc4-a37c-8b1907d9899f"). InnerVolumeSpecName "kube-api-access-6z726". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:28:18.187777 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.187715 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z726\" (UniqueName: \"kubernetes.io/projected/097695a0-bcfd-4dc4-a37c-8b1907d9899f-kube-api-access-6z726\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:28:18.187777 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.187738 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097695a0-bcfd-4dc4-a37c-8b1907d9899f-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:28:18.468374 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.468338 2559 generic.go:356] "Generic (PLEG): container finished" podID="097695a0-bcfd-4dc4-a37c-8b1907d9899f" containerID="835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb" exitCode=0 Sep 17 23:28:18.468531 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.468377 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" Sep 17 23:28:18.468531 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.468416 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" event={"ID":"097695a0-bcfd-4dc4-a37c-8b1907d9899f","Type":"ContainerDied","Data":"835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb"} Sep 17 23:28:18.468531 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.468468 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p" event={"ID":"097695a0-bcfd-4dc4-a37c-8b1907d9899f","Type":"ContainerDied","Data":"7239680bd02a53c041620af0faf077d7f03d435965877a00afc6b5357598abce"} Sep 17 23:28:18.468531 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.468490 2559 scope.go:117] "RemoveContainer" containerID="835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb" Sep 17 23:28:18.477342 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.477328 2559 scope.go:117] "RemoveContainer" containerID="835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb" Sep 17 23:28:18.477631 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:28:18.477612 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb\": container with ID starting with 835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb not found: ID does not exist" containerID="835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb" Sep 17 23:28:18.477677 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.477640 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb"} err="failed to get container status \"835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb\": rpc error: code = NotFound desc = could not find container \"835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb\": container with ID starting with 835b7903aa84974e2868c4023167dcfd6c76651ed475a8abe943d723405745bb not found: ID does not exist" Sep 17 23:28:18.483796 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.483776 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p"] Sep 17 23:28:18.487015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:18.486997 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-09450b8d-84b8bc6f86-qhr6p"] Sep 17 23:28:20.287482 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:20.287445 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="097695a0-bcfd-4dc4-a37c-8b1907d9899f" path="/var/lib/kubelet/pods/097695a0-bcfd-4dc4-a37c-8b1907d9899f/volumes" Sep 17 23:28:58.401889 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:28:58.401852 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:29:23.866606 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:23.866571 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762"] Sep 17 23:29:23.867086 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:23.866822 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" containerID="cri-o://c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c" gracePeriod=30 Sep 17 23:29:29.141891 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.141832 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:29:29.230046 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230013 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-data\") pod \"1768fa61-e9d1-4147-bddc-1a1c283335aa\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " Sep 17 23:29:29.230218 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230064 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-workload-socket\") pod \"1768fa61-e9d1-4147-bddc-1a1c283335aa\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " Sep 17 23:29:29.230218 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230095 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnjp7\" (UniqueName: \"kubernetes.io/projected/1768fa61-e9d1-4147-bddc-1a1c283335aa-kube-api-access-fnjp7\") pod \"1768fa61-e9d1-4147-bddc-1a1c283335aa\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " Sep 17 23:29:29.230218 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230119 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-envoy\") pod \"1768fa61-e9d1-4147-bddc-1a1c283335aa\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " Sep 17 23:29:29.230218 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230161 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-credential-socket\") pod \"1768fa61-e9d1-4147-bddc-1a1c283335aa\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " Sep 17 23:29:29.230218 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230208 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "1768fa61-e9d1-4147-bddc-1a1c283335aa" (UID: "1768fa61-e9d1-4147-bddc-1a1c283335aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:29:29.230476 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230243 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-workload-certs\") pod \"1768fa61-e9d1-4147-bddc-1a1c283335aa\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " Sep 17 23:29:29.230476 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230284 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1768fa61-e9d1-4147-bddc-1a1c283335aa" (UID: "1768fa61-e9d1-4147-bddc-1a1c283335aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:29:29.230476 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230296 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-podinfo\") pod \"1768fa61-e9d1-4147-bddc-1a1c283335aa\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " Sep 17 23:29:29.230476 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230342 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1768fa61-e9d1-4147-bddc-1a1c283335aa-istiod-ca-cert\") pod \"1768fa61-e9d1-4147-bddc-1a1c283335aa\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " Sep 17 23:29:29.230476 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230359 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-token\") pod \"1768fa61-e9d1-4147-bddc-1a1c283335aa\" (UID: \"1768fa61-e9d1-4147-bddc-1a1c283335aa\") " Sep 17 23:29:29.230714 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230488 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1768fa61-e9d1-4147-bddc-1a1c283335aa" (UID: "1768fa61-e9d1-4147-bddc-1a1c283335aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:29:29.230714 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230507 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1768fa61-e9d1-4147-bddc-1a1c283335aa" (UID: "1768fa61-e9d1-4147-bddc-1a1c283335aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:29:29.230875 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230855 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:29:29.230965 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230868 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1768fa61-e9d1-4147-bddc-1a1c283335aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1768fa61-e9d1-4147-bddc-1a1c283335aa" (UID: "1768fa61-e9d1-4147-bddc-1a1c283335aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:29:29.230965 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230881 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:29:29.230965 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230934 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:29:29.230965 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.230951 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:29:29.232367 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.232346 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1768fa61-e9d1-4147-bddc-1a1c283335aa" (UID: "1768fa61-e9d1-4147-bddc-1a1c283335aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:29:29.232466 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.232391 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1768fa61-e9d1-4147-bddc-1a1c283335aa" (UID: "1768fa61-e9d1-4147-bddc-1a1c283335aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:29:29.232466 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.232412 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1768fa61-e9d1-4147-bddc-1a1c283335aa-kube-api-access-fnjp7" (OuterVolumeSpecName: "kube-api-access-fnjp7") pod "1768fa61-e9d1-4147-bddc-1a1c283335aa" (UID: "1768fa61-e9d1-4147-bddc-1a1c283335aa"). InnerVolumeSpecName "kube-api-access-fnjp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:29:29.232466 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.232443 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "1768fa61-e9d1-4147-bddc-1a1c283335aa" (UID: "1768fa61-e9d1-4147-bddc-1a1c283335aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:29:29.332318 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.332290 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:29:29.332318 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.332313 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1768fa61-e9d1-4147-bddc-1a1c283335aa-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:29:29.332318 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.332322 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:29:29.332599 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.332330 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnjp7\" (UniqueName: \"kubernetes.io/projected/1768fa61-e9d1-4147-bddc-1a1c283335aa-kube-api-access-fnjp7\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:29:29.332599 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.332339 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1768fa61-e9d1-4147-bddc-1a1c283335aa-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:29:29.871526 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.871491 2559 generic.go:356] "Generic (PLEG): container finished" podID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerID="c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c" exitCode=0 Sep 17 23:29:29.871720 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.871534 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" Sep 17 23:29:29.871720 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.871579 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" event={"ID":"1768fa61-e9d1-4147-bddc-1a1c283335aa","Type":"ContainerDied","Data":"c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c"} Sep 17 23:29:29.871720 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.871643 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762" event={"ID":"1768fa61-e9d1-4147-bddc-1a1c283335aa","Type":"ContainerDied","Data":"b70b7e9600900454058c71742e6beb5240329ca6de9e0235547e0bde57c47d73"} Sep 17 23:29:29.871720 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.871675 2559 scope.go:117] "RemoveContainer" containerID="c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c" Sep 17 23:29:29.881979 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.881963 2559 scope.go:117] "RemoveContainer" containerID="90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f" Sep 17 23:29:29.890317 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.890298 2559 scope.go:117] "RemoveContainer" containerID="c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c" Sep 17 23:29:29.890555 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:29:29.890540 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c\": container with ID starting with c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c not found: ID does not exist" containerID="c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c" Sep 17 23:29:29.890612 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.890561 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c"} err="failed to get container status \"c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c\": rpc error: code = NotFound desc = could not find container \"c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c\": container with ID starting with c5cb562c3a0092d053fbcf93fd7a0292d15bca1921d184904a7fafa95800800c not found: ID does not exist" Sep 17 23:29:29.890612 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.890576 2559 scope.go:117] "RemoveContainer" containerID="90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f" Sep 17 23:29:29.890788 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:29:29.890770 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f\": container with ID starting with 90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f not found: ID does not exist" containerID="90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f" Sep 17 23:29:29.890834 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.890791 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f"} err="failed to get container status \"90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f\": rpc error: code = NotFound desc = could not find container \"90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f\": container with ID starting with 90fcda6666773db8feff1bf88255242c5c402ed8e54c4a762764f56f27af897f not found: ID does not exist" Sep 17 23:29:29.894242 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.894222 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762"] Sep 17 23:29:29.897911 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:29.897881 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-09450b8d-openshift-default-8579ff9cd4-p8762"] Sep 17 23:29:30.287491 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:30.287458 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" path="/var/lib/kubelet/pods/1768fa61-e9d1-4147-bddc-1a1c283335aa/volumes" Sep 17 23:29:33.311737 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:29:33.311700 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:30:00.154492 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.154399 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k"] Sep 17 23:30:00.154993 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.154869 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" Sep 17 23:30:00.154993 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.154883 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" Sep 17 23:30:00.154993 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.154937 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="097695a0-bcfd-4dc4-a37c-8b1907d9899f" containerName="payload-pre-processing" Sep 17 23:30:00.154993 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.154944 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="097695a0-bcfd-4dc4-a37c-8b1907d9899f" containerName="payload-pre-processing" Sep 17 23:30:00.155133 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.155006 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" Sep 17 23:30:00.155133 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.155017 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="097695a0-bcfd-4dc4-a37c-8b1907d9899f" containerName="payload-pre-processing" Sep 17 23:30:00.158329 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.158312 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7"] Sep 17 23:30:00.158443 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.158424 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k" Sep 17 23:30:00.158736 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.158717 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" Sep 17 23:30:00.158736 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.158737 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" Sep 17 23:30:00.158868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.158840 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1768fa61-e9d1-4147-bddc-1a1c283335aa" containerName="istio-proxy" Sep 17 23:30:00.161605 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.161587 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7" Sep 17 23:30:00.164033 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.164011 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d9szq\"" Sep 17 23:30:00.169643 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.169624 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k" Sep 17 23:30:00.174558 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.174215 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k"] Sep 17 23:30:00.177552 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.177526 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7" Sep 17 23:30:00.191641 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.191576 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7"] Sep 17 23:30:00.199140 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.199085 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf"] Sep 17 23:30:00.204839 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.204510 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf" Sep 17 23:30:00.212863 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.212841 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf"] Sep 17 23:30:00.220720 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.220629 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf" Sep 17 23:30:00.465521 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.465495 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k"] Sep 17 23:30:00.466818 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:30:00.466792 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6f3ef4_560d_405d_afa3_95194bf94640.slice/crio-8d26c58d6726d9645b587d8d6cd0c0fe0bf7a702164446811e3e82ee92d158bd WatchSource:0}: Error finding container 8d26c58d6726d9645b587d8d6cd0c0fe0bf7a702164446811e3e82ee92d158bd: Status 404 returned error can't find the container with id 8d26c58d6726d9645b587d8d6cd0c0fe0bf7a702164446811e3e82ee92d158bd Sep 17 23:30:00.479378 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.479360 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7"] Sep 17 23:30:00.481243 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:30:00.481221 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4275bb_fb03_49c9_9199_53601069a9a9.slice/crio-6116524d55b9f8059cdbdb242d71c49c836bc65acccef11b7a87ee1fc90abe3a WatchSource:0}: Error finding container 6116524d55b9f8059cdbdb242d71c49c836bc65acccef11b7a87ee1fc90abe3a: Status 404 returned error can't find the container with id 6116524d55b9f8059cdbdb242d71c49c836bc65acccef11b7a87ee1fc90abe3a Sep 17 23:30:00.517341 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:00.517314 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf"] Sep 17 23:30:00.519268 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:30:00.519244 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c30d6fc_c1e4_4a98_ae57_35030a10004a.slice/crio-838e151f9c912bc90d038842fd7ac4131ccf086b0520b244498ef91b1d8fa398 WatchSource:0}: Error finding container 838e151f9c912bc90d038842fd7ac4131ccf086b0520b244498ef91b1d8fa398: Status 404 returned error can't find the container with id 838e151f9c912bc90d038842fd7ac4131ccf086b0520b244498ef91b1d8fa398 Sep 17 23:30:01.050691 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:01.050584 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k" event={"ID":"db6f3ef4-560d-405d-afa3-95194bf94640","Type":"ContainerStarted","Data":"b084b2822ee4d7825c1d5d1d35c1b193799ad0a162797c1454d9ccbae3825fd6"} Sep 17 23:30:01.050691 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:01.050652 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k" event={"ID":"db6f3ef4-560d-405d-afa3-95194bf94640","Type":"ContainerStarted","Data":"8d26c58d6726d9645b587d8d6cd0c0fe0bf7a702164446811e3e82ee92d158bd"} Sep 17 23:30:01.052125 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:01.052096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7" event={"ID":"ba4275bb-fb03-49c9-9199-53601069a9a9","Type":"ContainerStarted","Data":"f841cd78390c51cf7019c1bf0ec1445b36ec05bbbbd71292d3e2f71d2e06c369"} Sep 17 23:30:01.052242 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:01.052148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7" event={"ID":"ba4275bb-fb03-49c9-9199-53601069a9a9","Type":"ContainerStarted","Data":"6116524d55b9f8059cdbdb242d71c49c836bc65acccef11b7a87ee1fc90abe3a"} Sep 17 23:30:01.053635 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:01.053613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf" event={"ID":"3c30d6fc-c1e4-4a98-ae57-35030a10004a","Type":"ContainerStarted","Data":"528aa07e3cb429ce2cfed864dd96b5b67f4d2b71e8bfbe6bb95f5824758a1679"} Sep 17 23:30:01.053732 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:01.053657 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf" event={"ID":"3c30d6fc-c1e4-4a98-ae57-35030a10004a","Type":"ContainerStarted","Data":"838e151f9c912bc90d038842fd7ac4131ccf086b0520b244498ef91b1d8fa398"} Sep 17 23:30:01.069096 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:01.069053 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k" podStartSLOduration=1.069035484 podStartE2EDuration="1.069035484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:01.067473986 +0000 UTC m=+1429.184953437" watchObservedRunningTime="2026-09-17 23:30:01.069035484 +0000 UTC m=+1429.186514934" Sep 17 23:30:01.082511 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:01.082473 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7" podStartSLOduration=1.082462329 podStartE2EDuration="1.082462329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:01.081497533 +0000 UTC m=+1429.198976982" watchObservedRunningTime="2026-09-17 23:30:01.082462329 +0000 UTC m=+1429.199941775" Sep 17 23:30:01.095717 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:01.095678 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf" podStartSLOduration=1.095666168 podStartE2EDuration="1.095666168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:01.09510137 +0000 UTC m=+1429.212580820" watchObservedRunningTime="2026-09-17 23:30:01.095666168 +0000 UTC m=+1429.213145618" Sep 17 23:30:02.060158 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:02.060127 2559 generic.go:356] "Generic (PLEG): container finished" podID="3c30d6fc-c1e4-4a98-ae57-35030a10004a" containerID="528aa07e3cb429ce2cfed864dd96b5b67f4d2b71e8bfbe6bb95f5824758a1679" exitCode=0 Sep 17 23:30:02.060576 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:02.060212 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf" event={"ID":"3c30d6fc-c1e4-4a98-ae57-35030a10004a","Type":"ContainerDied","Data":"528aa07e3cb429ce2cfed864dd96b5b67f4d2b71e8bfbe6bb95f5824758a1679"} Sep 17 23:30:02.061938 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:02.061887 2559 generic.go:356] "Generic (PLEG): container finished" podID="db6f3ef4-560d-405d-afa3-95194bf94640" containerID="b084b2822ee4d7825c1d5d1d35c1b193799ad0a162797c1454d9ccbae3825fd6" exitCode=0 Sep 17 23:30:02.062045 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:02.061974 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k" event={"ID":"db6f3ef4-560d-405d-afa3-95194bf94640","Type":"ContainerDied","Data":"b084b2822ee4d7825c1d5d1d35c1b193799ad0a162797c1454d9ccbae3825fd6"} Sep 17 23:30:02.063507 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:02.063488 2559 generic.go:356] "Generic (PLEG): container finished" podID="ba4275bb-fb03-49c9-9199-53601069a9a9" containerID="f841cd78390c51cf7019c1bf0ec1445b36ec05bbbbd71292d3e2f71d2e06c369" exitCode=0 Sep 17 23:30:02.063577 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:02.063518 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7" event={"ID":"ba4275bb-fb03-49c9-9199-53601069a9a9","Type":"ContainerDied","Data":"f841cd78390c51cf7019c1bf0ec1445b36ec05bbbbd71292d3e2f71d2e06c369"} Sep 17 23:30:03.276178 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:03.276155 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf" Sep 17 23:30:03.279508 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:03.279488 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k" Sep 17 23:30:03.283202 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:03.283184 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7" Sep 17 23:30:04.075995 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:04.075964 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7" Sep 17 23:30:04.075995 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:04.075986 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7" event={"ID":"ba4275bb-fb03-49c9-9199-53601069a9a9","Type":"ContainerDied","Data":"6116524d55b9f8059cdbdb242d71c49c836bc65acccef11b7a87ee1fc90abe3a"} Sep 17 23:30:04.076236 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:04.076034 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6116524d55b9f8059cdbdb242d71c49c836bc65acccef11b7a87ee1fc90abe3a" Sep 17 23:30:04.077311 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:04.077292 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf" Sep 17 23:30:04.077418 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:04.077312 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf" event={"ID":"3c30d6fc-c1e4-4a98-ae57-35030a10004a","Type":"ContainerDied","Data":"838e151f9c912bc90d038842fd7ac4131ccf086b0520b244498ef91b1d8fa398"} Sep 17 23:30:04.077418 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:04.077352 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="838e151f9c912bc90d038842fd7ac4131ccf086b0520b244498ef91b1d8fa398" Sep 17 23:30:04.078584 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:04.078566 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k" event={"ID":"db6f3ef4-560d-405d-afa3-95194bf94640","Type":"ContainerDied","Data":"8d26c58d6726d9645b587d8d6cd0c0fe0bf7a702164446811e3e82ee92d158bd"} Sep 17 23:30:04.078679 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:04.078587 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k" Sep 17 23:30:04.078679 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:04.078598 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d26c58d6726d9645b587d8d6cd0c0fe0bf7a702164446811e3e82ee92d158bd" Sep 17 23:30:10.112787 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:10.112757 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:30:18.257551 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.257504 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz"] Sep 17 23:30:18.258272 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.258245 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db6f3ef4-560d-405d-afa3-95194bf94640" containerName="cleanup" Sep 17 23:30:18.258272 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.258267 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="db6f3ef4-560d-405d-afa3-95194bf94640" containerName="cleanup" Sep 17 23:30:18.258439 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.258283 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba4275bb-fb03-49c9-9199-53601069a9a9" containerName="cleanup" Sep 17 23:30:18.258439 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.258297 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba4275bb-fb03-49c9-9199-53601069a9a9" containerName="cleanup" Sep 17 23:30:18.258439 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.258316 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c30d6fc-c1e4-4a98-ae57-35030a10004a" containerName="cleanup" Sep 17 23:30:18.258439 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.258324 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c30d6fc-c1e4-4a98-ae57-35030a10004a" containerName="cleanup" Sep 17 23:30:18.258639 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.258478 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba4275bb-fb03-49c9-9199-53601069a9a9" containerName="cleanup" Sep 17 23:30:18.258639 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.258493 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c30d6fc-c1e4-4a98-ae57-35030a10004a" containerName="cleanup" Sep 17 23:30:18.258639 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.258505 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="db6f3ef4-560d-405d-afa3-95194bf94640" containerName="cleanup" Sep 17 23:30:18.263001 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.262981 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.265700 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.265679 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9148ad-openshift-default-dockercfg-4zvf4\"" Sep 17 23:30:18.274365 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.274340 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz"] Sep 17 23:30:18.364588 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.364548 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrvv\" (UniqueName: \"kubernetes.io/projected/43e54ba1-48af-4cdb-a253-9e0f741c85a4-kube-api-access-hsrvv\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.364769 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.364605 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-data\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.364769 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.364713 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istiod-ca-cert\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.364769 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.364759 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-workload-socket\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.364953 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.364801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-envoy\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.364953 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.364832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-credential-socket\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.364953 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.364856 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-podinfo\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.365118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.364947 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-workload-certs\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.365118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.364982 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-token\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.466697 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.466643 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-workload-socket\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.466697 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.466705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-envoy\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.467015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.466730 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-credential-socket\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.467015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.466751 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-podinfo\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.467015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.466778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-workload-certs\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.467015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.466838 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-token\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.467015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.466953 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrvv\" (UniqueName: \"kubernetes.io/projected/43e54ba1-48af-4cdb-a253-9e0f741c85a4-kube-api-access-hsrvv\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.467015 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.466997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-data\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.467347 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.467051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istiod-ca-cert\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.467347 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.467263 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-workload-certs\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.467528 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.467506 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-workload-socket\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.467706 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.467629 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-data\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.467706 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.467689 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-credential-socket\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.467876 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.467853 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istiod-ca-cert\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.469165 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.469147 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-envoy\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.469664 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.469646 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-podinfo\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.476159 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.476137 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrvv\" (UniqueName: \"kubernetes.io/projected/43e54ba1-48af-4cdb-a253-9e0f741c85a4-kube-api-access-hsrvv\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.476282 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.476260 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-token\") pod \"e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.575962 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.575856 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:18.852321 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.852291 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz"] Sep 17 23:30:18.853758 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:30:18.853725 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e54ba1_48af_4cdb_a253_9e0f741c85a4.slice/crio-dd036c12769a7ce4f938818cf4dec9c66bafa8efa14160cf27cefdc311c7a076 WatchSource:0}: Error finding container dd036c12769a7ce4f938818cf4dec9c66bafa8efa14160cf27cefdc311c7a076: Status 404 returned error can't find the container with id dd036c12769a7ce4f938818cf4dec9c66bafa8efa14160cf27cefdc311c7a076 Sep 17 23:30:18.856082 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.856047 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:30:18.856165 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.856110 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:30:18.856165 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:18.856139 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:30:19.167699 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:19.167614 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" event={"ID":"43e54ba1-48af-4cdb-a253-9e0f741c85a4","Type":"ContainerStarted","Data":"63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba"} Sep 17 23:30:19.167699 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:19.167664 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" event={"ID":"43e54ba1-48af-4cdb-a253-9e0f741c85a4","Type":"ContainerStarted","Data":"dd036c12769a7ce4f938818cf4dec9c66bafa8efa14160cf27cefdc311c7a076"} Sep 17 23:30:19.193148 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:19.193096 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" podStartSLOduration=1.193081738 podStartE2EDuration="1.193081738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:19.191129069 +0000 UTC m=+1447.308608540" watchObservedRunningTime="2026-09-17 23:30:19.193081738 +0000 UTC m=+1447.310561258" Sep 17 23:30:19.576786 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:19.576756 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:19.582723 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:19.582697 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:20.173263 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:20.173230 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:20.174282 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:20.174253 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:30:26.518025 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.517991 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp"] Sep 17 23:30:26.521947 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.521926 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.524731 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.524712 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9148ad-serving-cert\"" Sep 17 23:30:26.532239 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.532217 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp"] Sep 17 23:30:26.644667 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.644611 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a36dcca-ae19-40c1-9769-3b8ee9102e56-openshift-service-ca\") pod \"maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.644850 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.644754 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a36dcca-ae19-40c1-9769-3b8ee9102e56-maas-api-tls\") pod \"maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.644850 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.644814 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knjmw\" (UniqueName: \"kubernetes.io/projected/7a36dcca-ae19-40c1-9769-3b8ee9102e56-kube-api-access-knjmw\") pod \"maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.644990 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.644854 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a36dcca-ae19-40c1-9769-3b8ee9102e56-metrics-certs\") pod \"maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.746479 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.746440 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a36dcca-ae19-40c1-9769-3b8ee9102e56-maas-api-tls\") pod \"maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.746661 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.746520 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knjmw\" (UniqueName: \"kubernetes.io/projected/7a36dcca-ae19-40c1-9769-3b8ee9102e56-kube-api-access-knjmw\") pod \"maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.746661 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.746566 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a36dcca-ae19-40c1-9769-3b8ee9102e56-metrics-certs\") pod \"maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.746780 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.746759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a36dcca-ae19-40c1-9769-3b8ee9102e56-openshift-service-ca\") pod \"maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.747432 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.747402 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a36dcca-ae19-40c1-9769-3b8ee9102e56-openshift-service-ca\") pod \"maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.749028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.749008 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a36dcca-ae19-40c1-9769-3b8ee9102e56-maas-api-tls\") pod \"maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.749161 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.749143 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a36dcca-ae19-40c1-9769-3b8ee9102e56-metrics-certs\") pod \"maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.755972 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.755948 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knjmw\" (UniqueName: \"kubernetes.io/projected/7a36dcca-ae19-40c1-9769-3b8ee9102e56-kube-api-access-knjmw\") pod \"maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:26.833048 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:26.832967 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:27.110062 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:27.110033 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp"] Sep 17 23:30:27.111683 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:30:27.111644 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a36dcca_ae19_40c1_9769_3b8ee9102e56.slice/crio-beb21fd948ad1c6dde1be04b1512dac28239cdcb94182a3c34dee0d3790f1566 WatchSource:0}: Error finding container beb21fd948ad1c6dde1be04b1512dac28239cdcb94182a3c34dee0d3790f1566: Status 404 returned error can't find the container with id beb21fd948ad1c6dde1be04b1512dac28239cdcb94182a3c34dee0d3790f1566 Sep 17 23:30:27.113035 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:27.113018 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 23:30:27.210839 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:27.210803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" event={"ID":"7a36dcca-ae19-40c1-9769-3b8ee9102e56","Type":"ContainerStarted","Data":"beb21fd948ad1c6dde1be04b1512dac28239cdcb94182a3c34dee0d3790f1566"} Sep 17 23:30:28.217734 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:28.217688 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" event={"ID":"7a36dcca-ae19-40c1-9769-3b8ee9102e56","Type":"ContainerStarted","Data":"2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a"} Sep 17 23:30:28.218225 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:28.217840 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:28.237106 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:28.237055 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" podStartSLOduration=1.830022166 podStartE2EDuration="2.237041884s" totalImagesPullingTime="407.019718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:28.235151517 +0000 UTC m=+1456.352630979" watchObservedRunningTime="2026-09-17 23:30:28.237041884 +0000 UTC m=+1456.354521334" Sep 17 23:30:29.518679 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:29.518586 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" podUID="572db407-342d-4115-aa58-fe567fc83eb9" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 17 23:30:30.231447 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:30.231408 2559 generic.go:356] "Generic (PLEG): container finished" podID="572db407-342d-4115-aa58-fe567fc83eb9" containerID="6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483" exitCode=137 Sep 17 23:30:30.231639 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:30.231483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" event={"ID":"572db407-342d-4115-aa58-fe567fc83eb9","Type":"ContainerDied","Data":"6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483"} Sep 17 23:30:30.231852 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:30.231836 2559 scope.go:117] "RemoveContainer" containerID="6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483" Sep 17 23:30:31.238887 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:31.238851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" event={"ID":"572db407-342d-4115-aa58-fe567fc83eb9","Type":"ContainerStarted","Data":"d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b"} Sep 17 23:30:31.239256 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:31.239057 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:30:34.229742 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:34.229712 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:30:42.247234 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:42.247196 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:30:51.977172 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:51.977135 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv"] Sep 17 23:30:51.981989 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:51.981964 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:51.984629 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:51.984578 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b54c32-openshift-default-dockercfg-58cqt\"" Sep 17 23:30:51.991027 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:51.991007 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv"] Sep 17 23:30:52.080669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.080626 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db5484f-87d6-4205-a721-19836998c3ed-istio-token\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.080860 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.080764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z4c9\" (UniqueName: \"kubernetes.io/projected/2db5484f-87d6-4205-a721-19836998c3ed-kube-api-access-6z4c9\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.080860 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.080822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-istio-data\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.081011 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.080883 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db5484f-87d6-4205-a721-19836998c3ed-istiod-ca-cert\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.081011 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.080938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db5484f-87d6-4205-a721-19836998c3ed-istio-podinfo\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.081011 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.080976 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-workload-socket\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.081011 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.081005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-istio-envoy\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.081221 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.081034 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-workload-certs\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.081221 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.081098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-credential-socket\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.181990 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.181959 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-istio-envoy\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.181990 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.181997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-workload-certs\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.182211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.182135 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-credential-socket\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.182211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.182193 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db5484f-87d6-4205-a721-19836998c3ed-istio-token\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.182310 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.182279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z4c9\" (UniqueName: \"kubernetes.io/projected/2db5484f-87d6-4205-a721-19836998c3ed-kube-api-access-6z4c9\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.182363 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.182338 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-istio-data\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.182451 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.182431 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-workload-certs\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.182508 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.182493 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-credential-socket\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.182686 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.182667 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-istio-data\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.182978 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.182961 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db5484f-87d6-4205-a721-19836998c3ed-istiod-ca-cert\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.183056 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.182986 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db5484f-87d6-4205-a721-19836998c3ed-istio-podinfo\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.183056 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.183014 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-workload-socket\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.183359 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.183337 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-workload-socket\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.184180 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.184156 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db5484f-87d6-4205-a721-19836998c3ed-istiod-ca-cert\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.184963 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.184938 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-istio-envoy\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.186021 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.186001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db5484f-87d6-4205-a721-19836998c3ed-istio-podinfo\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.190814 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.190793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z4c9\" (UniqueName: \"kubernetes.io/projected/2db5484f-87d6-4205-a721-19836998c3ed-kube-api-access-6z4c9\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.191059 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.191041 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db5484f-87d6-4205-a721-19836998c3ed-istio-token\") pod \"e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.295047 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.294973 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:52.581459 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.581435 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv"] Sep 17 23:30:52.583425 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:30:52.583393 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db5484f_87d6_4205_a721_19836998c3ed.slice/crio-1f05755d231742194a1a5f330c82f073c37b65bd9f11220db12989cb833ed129 WatchSource:0}: Error finding container 1f05755d231742194a1a5f330c82f073c37b65bd9f11220db12989cb833ed129: Status 404 returned error can't find the container with id 1f05755d231742194a1a5f330c82f073c37b65bd9f11220db12989cb833ed129 Sep 17 23:30:52.585714 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.585677 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:30:52.585824 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.585757 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:30:52.585824 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:52.585804 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:30:53.368724 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:53.368685 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" event={"ID":"2db5484f-87d6-4205-a721-19836998c3ed","Type":"ContainerStarted","Data":"aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80"} Sep 17 23:30:53.369194 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:53.368740 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" event={"ID":"2db5484f-87d6-4205-a721-19836998c3ed","Type":"ContainerStarted","Data":"1f05755d231742194a1a5f330c82f073c37b65bd9f11220db12989cb833ed129"} Sep 17 23:30:53.391834 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:53.391777 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" podStartSLOduration=2.391758054 podStartE2EDuration="2.391758054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:53.388754475 +0000 UTC m=+1481.506233922" watchObservedRunningTime="2026-09-17 23:30:53.391758054 +0000 UTC m=+1481.509237504" Sep 17 23:30:54.295665 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:54.295635 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:54.300839 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:54.300815 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:54.374244 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:54.374213 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:30:54.375231 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:30:54.375208 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:31:02.168923 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.167991 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5"] Sep 17 23:31:02.174294 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.174273 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" Sep 17 23:31:02.177135 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.177110 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b54c32\"" Sep 17 23:31:02.177254 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.177208 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b54c32-dockercfg-sdcnh\"" Sep 17 23:31:02.189688 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.189666 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5"] Sep 17 23:31:02.279595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.279560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sghlc\" (UniqueName: \"kubernetes.io/projected/a2283078-2189-4a3b-a4f0-6e0012aeb334-kube-api-access-sghlc\") pod \"payload-processing-e2e-api-b-b54c32-f695d495-vtwg5\" (UID: \"a2283078-2189-4a3b-a4f0-6e0012aeb334\") " pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" Sep 17 23:31:02.279770 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.279663 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2283078-2189-4a3b-a4f0-6e0012aeb334-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b54c32-f695d495-vtwg5\" (UID: \"a2283078-2189-4a3b-a4f0-6e0012aeb334\") " pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" Sep 17 23:31:02.380582 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.380548 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sghlc\" (UniqueName: \"kubernetes.io/projected/a2283078-2189-4a3b-a4f0-6e0012aeb334-kube-api-access-sghlc\") pod \"payload-processing-e2e-api-b-b54c32-f695d495-vtwg5\" (UID: \"a2283078-2189-4a3b-a4f0-6e0012aeb334\") " pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" Sep 17 23:31:02.380781 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.380623 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2283078-2189-4a3b-a4f0-6e0012aeb334-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b54c32-f695d495-vtwg5\" (UID: \"a2283078-2189-4a3b-a4f0-6e0012aeb334\") " pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" Sep 17 23:31:02.381334 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.381317 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2283078-2189-4a3b-a4f0-6e0012aeb334-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b54c32-f695d495-vtwg5\" (UID: \"a2283078-2189-4a3b-a4f0-6e0012aeb334\") " pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" Sep 17 23:31:02.388787 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.388758 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sghlc\" (UniqueName: \"kubernetes.io/projected/a2283078-2189-4a3b-a4f0-6e0012aeb334-kube-api-access-sghlc\") pod \"payload-processing-e2e-api-b-b54c32-f695d495-vtwg5\" (UID: \"a2283078-2189-4a3b-a4f0-6e0012aeb334\") " pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" Sep 17 23:31:02.485161 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.485133 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" Sep 17 23:31:02.769321 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:02.768819 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5"] Sep 17 23:31:02.771745 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:31:02.771709 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2283078_2189_4a3b_a4f0_6e0012aeb334.slice/crio-002d89d7547f7695c0599a73cffc1859835e29eb3f2caf4c3c20f6eb52685d90 WatchSource:0}: Error finding container 002d89d7547f7695c0599a73cffc1859835e29eb3f2caf4c3c20f6eb52685d90: Status 404 returned error can't find the container with id 002d89d7547f7695c0599a73cffc1859835e29eb3f2caf4c3c20f6eb52685d90 Sep 17 23:31:03.428554 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:03.428520 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" event={"ID":"a2283078-2189-4a3b-a4f0-6e0012aeb334","Type":"ContainerStarted","Data":"c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510"} Sep 17 23:31:03.428964 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:03.428587 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" event={"ID":"a2283078-2189-4a3b-a4f0-6e0012aeb334","Type":"ContainerStarted","Data":"002d89d7547f7695c0599a73cffc1859835e29eb3f2caf4c3c20f6eb52685d90"} Sep 17 23:31:03.428964 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:03.428612 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" Sep 17 23:31:03.445308 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:03.445257 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" podStartSLOduration=1.445216502 podStartE2EDuration="1.445216502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:31:03.444563933 +0000 UTC m=+1491.562043420" watchObservedRunningTime="2026-09-17 23:31:03.445216502 +0000 UTC m=+1491.562695953" Sep 17 23:31:09.466973 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:09.466939 2559 generic.go:356] "Generic (PLEG): container finished" podID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerID="63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba" exitCode=0 Sep 17 23:31:09.467376 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:09.467006 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" event={"ID":"43e54ba1-48af-4cdb-a253-9e0f741c85a4","Type":"ContainerDied","Data":"63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba"} Sep 17 23:31:09.467417 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:09.467407 2559 scope.go:117] "RemoveContainer" containerID="63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba" Sep 17 23:31:09.468590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:09.468565 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:31:09.468653 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:09.468640 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:31:09.468716 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:09.468703 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:31:10.474341 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:10.474306 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" event={"ID":"43e54ba1-48af-4cdb-a253-9e0f741c85a4","Type":"ContainerStarted","Data":"a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24"} Sep 17 23:31:10.576850 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:10.576815 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:31:10.577084 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:10.577051 2559 patch_prober.go:28] interesting pod/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 17 23:31:10.577133 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:10.577098 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 17 23:31:11.576603 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:11.576565 2559 patch_prober.go:28] interesting pod/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 17 23:31:11.577118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:11.576641 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 17 23:31:12.576922 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:12.576859 2559 patch_prober.go:28] interesting pod/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 17 23:31:12.577316 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:12.576959 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 17 23:31:13.576433 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:13.576396 2559 patch_prober.go:28] interesting pod/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 17 23:31:13.576612 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:13.576451 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 17 23:31:14.300017 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:14.299991 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:31:14.303791 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:14.303770 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:31:14.436682 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:14.436652 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" Sep 17 23:31:14.576690 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:14.576590 2559 patch_prober.go:28] interesting pod/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 17 23:31:14.576690 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:14.576657 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 17 23:31:15.576518 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:15.576481 2559 patch_prober.go:28] interesting pod/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 17 23:31:15.576892 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:15.576554 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 17 23:31:16.576714 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:16.576674 2559 patch_prober.go:28] interesting pod/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 17 23:31:16.577150 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:16.576737 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 17 23:31:17.581213 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:17.581177 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:31:17.581590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:17.581466 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:31:17.582066 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:17.582046 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:31:17.607583 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:17.607549 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:31:28.590926 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:28.590824 2559 generic.go:356] "Generic (PLEG): container finished" podID="2db5484f-87d6-4205-a721-19836998c3ed" containerID="aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80" exitCode=0 Sep 17 23:31:28.591412 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:28.590925 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" event={"ID":"2db5484f-87d6-4205-a721-19836998c3ed","Type":"ContainerDied","Data":"aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80"} Sep 17 23:31:28.591486 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:28.591469 2559 scope.go:117] "RemoveContainer" containerID="aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80" Sep 17 23:31:28.592718 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:28.592688 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:31:28.592802 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:28.592788 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:31:28.592838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:28.592824 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:31:29.603391 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:29.603356 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" event={"ID":"2db5484f-87d6-4205-a721-19836998c3ed","Type":"ContainerStarted","Data":"9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223"} Sep 17 23:31:30.295498 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:30.295447 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:31:30.295776 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:30.295746 2559 patch_prober.go:28] interesting pod/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 17 23:31:30.295940 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:30.295878 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" podUID="2db5484f-87d6-4205-a721-19836998c3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 17 23:31:31.296028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:31.295997 2559 patch_prober.go:28] interesting pod/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 17 23:31:31.296408 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:31.296062 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" podUID="2db5484f-87d6-4205-a721-19836998c3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 17 23:31:32.295349 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:32.295316 2559 patch_prober.go:28] interesting pod/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 17 23:31:32.295592 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:32.295384 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" podUID="2db5484f-87d6-4205-a721-19836998c3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 17 23:31:33.295693 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:33.295664 2559 patch_prober.go:28] interesting pod/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 17 23:31:33.296077 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:33.295717 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" podUID="2db5484f-87d6-4205-a721-19836998c3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 17 23:31:34.295323 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:34.295289 2559 patch_prober.go:28] interesting pod/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 17 23:31:34.295524 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:34.295346 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" podUID="2db5484f-87d6-4205-a721-19836998c3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 17 23:31:35.302875 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:35.302845 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:31:35.303336 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:35.303194 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:31:35.304324 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:35.304301 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:31:43.639389 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:43.639350 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp"] Sep 17 23:31:43.639836 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:43.639646 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" podUID="7a36dcca-ae19-40c1-9769-3b8ee9102e56" containerName="maas-api" containerID="cri-o://2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a" gracePeriod=30 Sep 17 23:31:43.901519 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:43.901464 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:31:44.020682 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.020651 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a36dcca-ae19-40c1-9769-3b8ee9102e56-openshift-service-ca\") pod \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " Sep 17 23:31:44.020868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.020703 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a36dcca-ae19-40c1-9769-3b8ee9102e56-maas-api-tls\") pod \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " Sep 17 23:31:44.020868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.020782 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knjmw\" (UniqueName: \"kubernetes.io/projected/7a36dcca-ae19-40c1-9769-3b8ee9102e56-kube-api-access-knjmw\") pod \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " Sep 17 23:31:44.020868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.020845 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a36dcca-ae19-40c1-9769-3b8ee9102e56-metrics-certs\") pod \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\" (UID: \"7a36dcca-ae19-40c1-9769-3b8ee9102e56\") " Sep 17 23:31:44.021064 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.021015 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a36dcca-ae19-40c1-9769-3b8ee9102e56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a36dcca-ae19-40c1-9769-3b8ee9102e56" (UID: "7a36dcca-ae19-40c1-9769-3b8ee9102e56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:31:44.021241 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.021216 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a36dcca-ae19-40c1-9769-3b8ee9102e56-openshift-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:44.023009 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.022982 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a36dcca-ae19-40c1-9769-3b8ee9102e56-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a36dcca-ae19-40c1-9769-3b8ee9102e56" (UID: "7a36dcca-ae19-40c1-9769-3b8ee9102e56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:31:44.023117 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.023050 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a36dcca-ae19-40c1-9769-3b8ee9102e56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a36dcca-ae19-40c1-9769-3b8ee9102e56" (UID: "7a36dcca-ae19-40c1-9769-3b8ee9102e56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:31:44.023268 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.023250 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a36dcca-ae19-40c1-9769-3b8ee9102e56-kube-api-access-knjmw" (OuterVolumeSpecName: "kube-api-access-knjmw") pod "7a36dcca-ae19-40c1-9769-3b8ee9102e56" (UID: "7a36dcca-ae19-40c1-9769-3b8ee9102e56"). InnerVolumeSpecName "kube-api-access-knjmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:31:44.122419 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.122371 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knjmw\" (UniqueName: \"kubernetes.io/projected/7a36dcca-ae19-40c1-9769-3b8ee9102e56-kube-api-access-knjmw\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:44.122419 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.122405 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a36dcca-ae19-40c1-9769-3b8ee9102e56-metrics-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:44.122419 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.122414 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a36dcca-ae19-40c1-9769-3b8ee9102e56-maas-api-tls\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:44.692743 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.692652 2559 generic.go:356] "Generic (PLEG): container finished" podID="7a36dcca-ae19-40c1-9769-3b8ee9102e56" containerID="2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a" exitCode=0 Sep 17 23:31:44.692743 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.692694 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" Sep 17 23:31:44.693193 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.692734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" event={"ID":"7a36dcca-ae19-40c1-9769-3b8ee9102e56","Type":"ContainerDied","Data":"2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a"} Sep 17 23:31:44.693193 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.692787 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp" event={"ID":"7a36dcca-ae19-40c1-9769-3b8ee9102e56","Type":"ContainerDied","Data":"beb21fd948ad1c6dde1be04b1512dac28239cdcb94182a3c34dee0d3790f1566"} Sep 17 23:31:44.693193 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.692808 2559 scope.go:117] "RemoveContainer" containerID="2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a" Sep 17 23:31:44.701857 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.701841 2559 scope.go:117] "RemoveContainer" containerID="2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a" Sep 17 23:31:44.702241 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:31:44.702224 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a\": container with ID starting with 2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a not found: ID does not exist" containerID="2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a" Sep 17 23:31:44.702292 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.702251 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a"} err="failed to get container status \"2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a\": rpc error: code = NotFound desc = could not find container \"2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a\": container with ID starting with 2f3c0ec0883448a04948d7910deddd0fb3d0a8cd3178ae9f0b6157312f66f85a not found: ID does not exist" Sep 17 23:31:44.713038 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.713018 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp"] Sep 17 23:31:44.717066 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:44.717043 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9148ad-7464bc4755-rxnlp"] Sep 17 23:31:46.288609 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:46.288576 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a36dcca-ae19-40c1-9769-3b8ee9102e56" path="/var/lib/kubelet/pods/7a36dcca-ae19-40c1-9769-3b8ee9102e56/volumes" Sep 17 23:31:50.358620 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:50.358591 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz"] Sep 17 23:31:50.359038 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:50.358801 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" containerID="cri-o://a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24" gracePeriod=30 Sep 17 23:31:55.616700 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.616676 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:31:55.740946 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.740883 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-workload-socket\") pod \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " Sep 17 23:31:55.741242 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741008 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-workload-certs\") pod \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " Sep 17 23:31:55.741242 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741034 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-data\") pod \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " Sep 17 23:31:55.741242 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741093 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-envoy\") pod \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " Sep 17 23:31:55.741242 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741117 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-token\") pod \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " Sep 17 23:31:55.741242 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741143 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "43e54ba1-48af-4cdb-a253-9e0f741c85a4" (UID: "43e54ba1-48af-4cdb-a253-9e0f741c85a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:31:55.741242 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741152 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-credential-socket\") pod \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " Sep 17 23:31:55.741242 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741235 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsrvv\" (UniqueName: \"kubernetes.io/projected/43e54ba1-48af-4cdb-a253-9e0f741c85a4-kube-api-access-hsrvv\") pod \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " Sep 17 23:31:55.741628 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741270 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-podinfo\") pod \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " Sep 17 23:31:55.741628 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741309 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istiod-ca-cert\") pod \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\" (UID: \"43e54ba1-48af-4cdb-a253-9e0f741c85a4\") " Sep 17 23:31:55.741628 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741343 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "43e54ba1-48af-4cdb-a253-9e0f741c85a4" (UID: "43e54ba1-48af-4cdb-a253-9e0f741c85a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:31:55.741628 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741377 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "43e54ba1-48af-4cdb-a253-9e0f741c85a4" (UID: "43e54ba1-48af-4cdb-a253-9e0f741c85a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:31:55.741841 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741713 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:55.741841 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741732 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:55.741841 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741746 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:55.741841 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741740 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "43e54ba1-48af-4cdb-a253-9e0f741c85a4" (UID: "43e54ba1-48af-4cdb-a253-9e0f741c85a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:31:55.742070 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.741855 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "43e54ba1-48af-4cdb-a253-9e0f741c85a4" (UID: "43e54ba1-48af-4cdb-a253-9e0f741c85a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:31:55.743565 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.743540 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "43e54ba1-48af-4cdb-a253-9e0f741c85a4" (UID: "43e54ba1-48af-4cdb-a253-9e0f741c85a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:31:55.743889 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.743857 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "43e54ba1-48af-4cdb-a253-9e0f741c85a4" (UID: "43e54ba1-48af-4cdb-a253-9e0f741c85a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:31:55.744027 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.743969 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "43e54ba1-48af-4cdb-a253-9e0f741c85a4" (UID: "43e54ba1-48af-4cdb-a253-9e0f741c85a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:31:55.744027 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.743970 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e54ba1-48af-4cdb-a253-9e0f741c85a4-kube-api-access-hsrvv" (OuterVolumeSpecName: "kube-api-access-hsrvv") pod "43e54ba1-48af-4cdb-a253-9e0f741c85a4" (UID: "43e54ba1-48af-4cdb-a253-9e0f741c85a4"). InnerVolumeSpecName "kube-api-access-hsrvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:31:55.766117 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.766087 2559 generic.go:356] "Generic (PLEG): container finished" podID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerID="a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24" exitCode=0 Sep 17 23:31:55.766220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.766133 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" Sep 17 23:31:55.766220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.766150 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" event={"ID":"43e54ba1-48af-4cdb-a253-9e0f741c85a4","Type":"ContainerDied","Data":"a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24"} Sep 17 23:31:55.766220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.766199 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz" event={"ID":"43e54ba1-48af-4cdb-a253-9e0f741c85a4","Type":"ContainerDied","Data":"dd036c12769a7ce4f938818cf4dec9c66bafa8efa14160cf27cefdc311c7a076"} Sep 17 23:31:55.766403 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.766230 2559 scope.go:117] "RemoveContainer" containerID="a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24" Sep 17 23:31:55.775521 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.775506 2559 scope.go:117] "RemoveContainer" containerID="63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba" Sep 17 23:31:55.783719 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.783699 2559 scope.go:117] "RemoveContainer" containerID="a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24" Sep 17 23:31:55.784161 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:31:55.784107 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24\": container with ID starting with a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24 not found: ID does not exist" containerID="a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24" Sep 17 23:31:55.784269 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.784192 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24"} err="failed to get container status \"a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24\": rpc error: code = NotFound desc = could not find container \"a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24\": container with ID starting with a6c0b53b96655eabb0d3fd2b450ecfc41cc5f5cf02f3aa59837333764ad0ca24 not found: ID does not exist" Sep 17 23:31:55.784269 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.784232 2559 scope.go:117] "RemoveContainer" containerID="63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba" Sep 17 23:31:55.784562 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:31:55.784532 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba\": container with ID starting with 63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba not found: ID does not exist" containerID="63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba" Sep 17 23:31:55.784665 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.784571 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba"} err="failed to get container status \"63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba\": rpc error: code = NotFound desc = could not find container \"63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba\": container with ID starting with 63cd60d13853f841f1a5bd7f116fa6798be2086a79fe6b6607555e8cf4ac90ba not found: ID does not exist" Sep 17 23:31:55.787181 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.787163 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz"] Sep 17 23:31:55.790741 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.790719 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9148ad-openshift-default-7c7df558c8-m8dbz"] Sep 17 23:31:55.843164 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.843134 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:55.843164 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.843158 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:55.843164 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.843166 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:55.843421 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.843176 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:55.843421 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.843183 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsrvv\" (UniqueName: \"kubernetes.io/projected/43e54ba1-48af-4cdb-a253-9e0f741c85a4-kube-api-access-hsrvv\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:55.843421 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:55.843192 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e54ba1-48af-4cdb-a253-9e0f741c85a4-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:31:56.288718 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:31:56.288688 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" path="/var/lib/kubelet/pods/43e54ba1-48af-4cdb-a253-9e0f741c85a4/volumes" Sep 17 23:32:01.102886 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:01.102853 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:32:06.316191 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.316157 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5"] Sep 17 23:32:06.316692 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.316373 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" podUID="a2283078-2189-4a3b-a4f0-6e0012aeb334" containerName="payload-processing" containerID="cri-o://c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510" gracePeriod=30 Sep 17 23:32:06.589844 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.589821 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" Sep 17 23:32:06.648465 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.648055 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sghlc\" (UniqueName: \"kubernetes.io/projected/a2283078-2189-4a3b-a4f0-6e0012aeb334-kube-api-access-sghlc\") pod \"a2283078-2189-4a3b-a4f0-6e0012aeb334\" (UID: \"a2283078-2189-4a3b-a4f0-6e0012aeb334\") " Sep 17 23:32:06.648465 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.648156 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2283078-2189-4a3b-a4f0-6e0012aeb334-plugins-config-volume\") pod \"a2283078-2189-4a3b-a4f0-6e0012aeb334\" (UID: \"a2283078-2189-4a3b-a4f0-6e0012aeb334\") " Sep 17 23:32:06.650150 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.648723 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2283078-2189-4a3b-a4f0-6e0012aeb334-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2283078-2189-4a3b-a4f0-6e0012aeb334" (UID: "a2283078-2189-4a3b-a4f0-6e0012aeb334"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:32:06.651295 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.651255 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2283078-2189-4a3b-a4f0-6e0012aeb334-kube-api-access-sghlc" (OuterVolumeSpecName: "kube-api-access-sghlc") pod "a2283078-2189-4a3b-a4f0-6e0012aeb334" (UID: "a2283078-2189-4a3b-a4f0-6e0012aeb334"). InnerVolumeSpecName "kube-api-access-sghlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:06.749263 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.749224 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2283078-2189-4a3b-a4f0-6e0012aeb334-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:06.749263 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.749256 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sghlc\" (UniqueName: \"kubernetes.io/projected/a2283078-2189-4a3b-a4f0-6e0012aeb334-kube-api-access-sghlc\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:06.829104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.829018 2559 generic.go:356] "Generic (PLEG): container finished" podID="a2283078-2189-4a3b-a4f0-6e0012aeb334" containerID="c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510" exitCode=0 Sep 17 23:32:06.829104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.829069 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" Sep 17 23:32:06.829300 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.829103 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" event={"ID":"a2283078-2189-4a3b-a4f0-6e0012aeb334","Type":"ContainerDied","Data":"c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510"} Sep 17 23:32:06.829300 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.829160 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5" event={"ID":"a2283078-2189-4a3b-a4f0-6e0012aeb334","Type":"ContainerDied","Data":"002d89d7547f7695c0599a73cffc1859835e29eb3f2caf4c3c20f6eb52685d90"} Sep 17 23:32:06.829300 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.829181 2559 scope.go:117] "RemoveContainer" containerID="c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510" Sep 17 23:32:06.838399 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.838383 2559 scope.go:117] "RemoveContainer" containerID="c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510" Sep 17 23:32:06.838665 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:32:06.838643 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510\": container with ID starting with c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510 not found: ID does not exist" containerID="c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510" Sep 17 23:32:06.838717 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.838673 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510"} err="failed to get container status \"c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510\": rpc error: code = NotFound desc = could not find container \"c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510\": container with ID starting with c49b1d0f35ca6a3c6203990738ebe656035816311c32b27f12f584e0b397a510 not found: ID does not exist" Sep 17 23:32:06.851133 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.851106 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5"] Sep 17 23:32:06.855061 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:06.855037 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b54c32-f695d495-vtwg5"] Sep 17 23:32:08.288252 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:08.288218 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2283078-2189-4a3b-a4f0-6e0012aeb334" path="/var/lib/kubelet/pods/a2283078-2189-4a3b-a4f0-6e0012aeb334/volumes" Sep 17 23:32:13.178751 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:13.178720 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv"] Sep 17 23:32:13.179201 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:13.178971 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" podUID="2db5484f-87d6-4205-a721-19836998c3ed" containerName="istio-proxy" containerID="cri-o://9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223" gracePeriod=30 Sep 17 23:32:18.440433 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.440369 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:32:18.560333 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560304 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-workload-certs\") pod \"2db5484f-87d6-4205-a721-19836998c3ed\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " Sep 17 23:32:18.560333 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560335 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-credential-socket\") pod \"2db5484f-87d6-4205-a721-19836998c3ed\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " Sep 17 23:32:18.560575 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560359 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-istio-data\") pod \"2db5484f-87d6-4205-a721-19836998c3ed\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " Sep 17 23:32:18.560575 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560396 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db5484f-87d6-4205-a721-19836998c3ed-istiod-ca-cert\") pod \"2db5484f-87d6-4205-a721-19836998c3ed\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " Sep 17 23:32:18.560575 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560455 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z4c9\" (UniqueName: \"kubernetes.io/projected/2db5484f-87d6-4205-a721-19836998c3ed-kube-api-access-6z4c9\") pod \"2db5484f-87d6-4205-a721-19836998c3ed\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " Sep 17 23:32:18.560575 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560474 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db5484f-87d6-4205-a721-19836998c3ed-istio-token\") pod \"2db5484f-87d6-4205-a721-19836998c3ed\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " Sep 17 23:32:18.560575 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560567 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-workload-socket\") pod \"2db5484f-87d6-4205-a721-19836998c3ed\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " Sep 17 23:32:18.560827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560601 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db5484f-87d6-4205-a721-19836998c3ed-istio-podinfo\") pod \"2db5484f-87d6-4205-a721-19836998c3ed\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " Sep 17 23:32:18.560827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560670 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-istio-envoy\") pod \"2db5484f-87d6-4205-a721-19836998c3ed\" (UID: \"2db5484f-87d6-4205-a721-19836998c3ed\") " Sep 17 23:32:18.560827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560666 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2db5484f-87d6-4205-a721-19836998c3ed" (UID: "2db5484f-87d6-4205-a721-19836998c3ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:18.560827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560707 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2db5484f-87d6-4205-a721-19836998c3ed" (UID: "2db5484f-87d6-4205-a721-19836998c3ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:18.560827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560725 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "2db5484f-87d6-4205-a721-19836998c3ed" (UID: "2db5484f-87d6-4205-a721-19836998c3ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:18.561105 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.560850 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2db5484f-87d6-4205-a721-19836998c3ed" (UID: "2db5484f-87d6-4205-a721-19836998c3ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:18.561105 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.561016 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:18.561105 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.561037 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:18.561105 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.561049 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:18.561105 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.561085 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:18.561290 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.561217 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db5484f-87d6-4205-a721-19836998c3ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2db5484f-87d6-4205-a721-19836998c3ed" (UID: "2db5484f-87d6-4205-a721-19836998c3ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:32:18.562719 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.562690 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db5484f-87d6-4205-a721-19836998c3ed-kube-api-access-6z4c9" (OuterVolumeSpecName: "kube-api-access-6z4c9") pod "2db5484f-87d6-4205-a721-19836998c3ed" (UID: "2db5484f-87d6-4205-a721-19836998c3ed"). InnerVolumeSpecName "kube-api-access-6z4c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:18.562834 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.562743 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db5484f-87d6-4205-a721-19836998c3ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "2db5484f-87d6-4205-a721-19836998c3ed" (UID: "2db5484f-87d6-4205-a721-19836998c3ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:18.563119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.563094 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2db5484f-87d6-4205-a721-19836998c3ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2db5484f-87d6-4205-a721-19836998c3ed" (UID: "2db5484f-87d6-4205-a721-19836998c3ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:32:18.563119 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.563100 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2db5484f-87d6-4205-a721-19836998c3ed" (UID: "2db5484f-87d6-4205-a721-19836998c3ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:18.661607 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.661576 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z4c9\" (UniqueName: \"kubernetes.io/projected/2db5484f-87d6-4205-a721-19836998c3ed-kube-api-access-6z4c9\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:18.661607 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.661599 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db5484f-87d6-4205-a721-19836998c3ed-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:18.661607 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.661608 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db5484f-87d6-4205-a721-19836998c3ed-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:18.661607 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.661615 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db5484f-87d6-4205-a721-19836998c3ed-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:18.661849 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.661624 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db5484f-87d6-4205-a721-19836998c3ed-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:18.900471 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.900435 2559 generic.go:356] "Generic (PLEG): container finished" podID="2db5484f-87d6-4205-a721-19836998c3ed" containerID="9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223" exitCode=0 Sep 17 23:32:18.900662 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.900484 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" event={"ID":"2db5484f-87d6-4205-a721-19836998c3ed","Type":"ContainerDied","Data":"9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223"} Sep 17 23:32:18.900662 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.900486 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" Sep 17 23:32:18.900662 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.900525 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv" event={"ID":"2db5484f-87d6-4205-a721-19836998c3ed","Type":"ContainerDied","Data":"1f05755d231742194a1a5f330c82f073c37b65bd9f11220db12989cb833ed129"} Sep 17 23:32:18.900662 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.900546 2559 scope.go:117] "RemoveContainer" containerID="9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223" Sep 17 23:32:18.911175 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.911153 2559 scope.go:117] "RemoveContainer" containerID="aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80" Sep 17 23:32:18.919390 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.919372 2559 scope.go:117] "RemoveContainer" containerID="9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223" Sep 17 23:32:18.919661 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:32:18.919643 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223\": container with ID starting with 9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223 not found: ID does not exist" containerID="9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223" Sep 17 23:32:18.919706 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.919670 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223"} err="failed to get container status \"9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223\": rpc error: code = NotFound desc = could not find container \"9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223\": container with ID starting with 9047b6e5878d8c64d8881d76b320a0cdb9cb8edc29f6c3bca25dea34b6a35223 not found: ID does not exist" Sep 17 23:32:18.919706 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.919689 2559 scope.go:117] "RemoveContainer" containerID="aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80" Sep 17 23:32:18.919949 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:32:18.919935 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80\": container with ID starting with aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80 not found: ID does not exist" containerID="aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80" Sep 17 23:32:18.919995 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.919954 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80"} err="failed to get container status \"aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80\": rpc error: code = NotFound desc = could not find container \"aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80\": container with ID starting with aa315d73bef61e1b6c4765db2adcfb69797f6f4db00baac282f64fd72e9d8f80 not found: ID does not exist" Sep 17 23:32:18.925827 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.925805 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv"] Sep 17 23:32:18.929034 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:18.929007 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b54c32-openshift-default-5f7c977d84-z85wv"] Sep 17 23:32:20.287658 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:20.287625 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2db5484f-87d6-4205-a721-19836998c3ed" path="/var/lib/kubelet/pods/2db5484f-87d6-4205-a721-19836998c3ed/volumes" Sep 17 23:32:24.004162 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.004125 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv"] Sep 17 23:32:24.004636 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.004422 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" podUID="45d8d906-d030-4731-94ac-3522ad0b41bb" containerName="payload-processing" containerID="cri-o://0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c" gracePeriod=30 Sep 17 23:32:24.014061 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.014036 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh"] Sep 17 23:32:24.014295 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.014244 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" podUID="d67396b8-6a45-456f-b541-45defaca6bf6" containerName="payload-pre-processing" containerID="cri-o://ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce" gracePeriod=30 Sep 17 23:32:24.293151 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.293128 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" Sep 17 23:32:24.296124 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.296102 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" Sep 17 23:32:24.419446 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.419408 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsnzb\" (UniqueName: \"kubernetes.io/projected/d67396b8-6a45-456f-b541-45defaca6bf6-kube-api-access-zsnzb\") pod \"d67396b8-6a45-456f-b541-45defaca6bf6\" (UID: \"d67396b8-6a45-456f-b541-45defaca6bf6\") " Sep 17 23:32:24.419633 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.419488 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d8d906-d030-4731-94ac-3522ad0b41bb-plugins-config-volume\") pod \"45d8d906-d030-4731-94ac-3522ad0b41bb\" (UID: \"45d8d906-d030-4731-94ac-3522ad0b41bb\") " Sep 17 23:32:24.419633 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.419545 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c828s\" (UniqueName: \"kubernetes.io/projected/45d8d906-d030-4731-94ac-3522ad0b41bb-kube-api-access-c828s\") pod \"45d8d906-d030-4731-94ac-3522ad0b41bb\" (UID: \"45d8d906-d030-4731-94ac-3522ad0b41bb\") " Sep 17 23:32:24.419633 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.419580 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67396b8-6a45-456f-b541-45defaca6bf6-plugins-config-volume\") pod \"d67396b8-6a45-456f-b541-45defaca6bf6\" (UID: \"d67396b8-6a45-456f-b541-45defaca6bf6\") " Sep 17 23:32:24.420036 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.419999 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d8d906-d030-4731-94ac-3522ad0b41bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45d8d906-d030-4731-94ac-3522ad0b41bb" (UID: "45d8d906-d030-4731-94ac-3522ad0b41bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:32:24.420178 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.420058 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67396b8-6a45-456f-b541-45defaca6bf6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d67396b8-6a45-456f-b541-45defaca6bf6" (UID: "d67396b8-6a45-456f-b541-45defaca6bf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:32:24.421725 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.421700 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d8d906-d030-4731-94ac-3522ad0b41bb-kube-api-access-c828s" (OuterVolumeSpecName: "kube-api-access-c828s") pod "45d8d906-d030-4731-94ac-3522ad0b41bb" (UID: "45d8d906-d030-4731-94ac-3522ad0b41bb"). InnerVolumeSpecName "kube-api-access-c828s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:24.421817 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.421708 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67396b8-6a45-456f-b541-45defaca6bf6-kube-api-access-zsnzb" (OuterVolumeSpecName: "kube-api-access-zsnzb") pod "d67396b8-6a45-456f-b541-45defaca6bf6" (UID: "d67396b8-6a45-456f-b541-45defaca6bf6"). InnerVolumeSpecName "kube-api-access-zsnzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:24.520493 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.520409 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d8d906-d030-4731-94ac-3522ad0b41bb-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:24.520493 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.520439 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c828s\" (UniqueName: \"kubernetes.io/projected/45d8d906-d030-4731-94ac-3522ad0b41bb-kube-api-access-c828s\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:24.520493 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.520450 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67396b8-6a45-456f-b541-45defaca6bf6-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:24.520493 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.520458 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsnzb\" (UniqueName: \"kubernetes.io/projected/d67396b8-6a45-456f-b541-45defaca6bf6-kube-api-access-zsnzb\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:24.935211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.935122 2559 generic.go:356] "Generic (PLEG): container finished" podID="d67396b8-6a45-456f-b541-45defaca6bf6" containerID="ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce" exitCode=0 Sep 17 23:32:24.935211 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.935165 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" Sep 17 23:32:24.935411 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.935211 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" event={"ID":"d67396b8-6a45-456f-b541-45defaca6bf6","Type":"ContainerDied","Data":"ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce"} Sep 17 23:32:24.935411 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.935266 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh" event={"ID":"d67396b8-6a45-456f-b541-45defaca6bf6","Type":"ContainerDied","Data":"7fac441904497083605f2ed68b9eca6c00d86635b59a1bb1ef80b8f2625a9733"} Sep 17 23:32:24.935411 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.935289 2559 scope.go:117] "RemoveContainer" containerID="ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce" Sep 17 23:32:24.936628 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.936600 2559 generic.go:356] "Generic (PLEG): container finished" podID="45d8d906-d030-4731-94ac-3522ad0b41bb" containerID="0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c" exitCode=0 Sep 17 23:32:24.936705 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.936655 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" Sep 17 23:32:24.936705 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.936666 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" event={"ID":"45d8d906-d030-4731-94ac-3522ad0b41bb","Type":"ContainerDied","Data":"0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c"} Sep 17 23:32:24.936783 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.936723 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv" event={"ID":"45d8d906-d030-4731-94ac-3522ad0b41bb","Type":"ContainerDied","Data":"495e453ab2d7ceb985bbc42b95ecc0bdbe20b044c329711fffbaf95e2b9efa2f"} Sep 17 23:32:24.949743 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.949725 2559 scope.go:117] "RemoveContainer" containerID="ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce" Sep 17 23:32:24.950204 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:32:24.950178 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce\": container with ID starting with ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce not found: ID does not exist" containerID="ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce" Sep 17 23:32:24.950294 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.950214 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce"} err="failed to get container status \"ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce\": rpc error: code = NotFound desc = could not find container \"ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce\": container with ID starting with ba53c0cd2662c1ff64afaa22f8298e0d23ef9a555c0b5baf8b940f1b863b0bce not found: ID does not exist" Sep 17 23:32:24.950294 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.950237 2559 scope.go:117] "RemoveContainer" containerID="0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c" Sep 17 23:32:24.958860 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.958843 2559 scope.go:117] "RemoveContainer" containerID="0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c" Sep 17 23:32:24.959128 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:32:24.959111 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c\": container with ID starting with 0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c not found: ID does not exist" containerID="0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c" Sep 17 23:32:24.959180 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.959134 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c"} err="failed to get container status \"0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c\": rpc error: code = NotFound desc = could not find container \"0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c\": container with ID starting with 0d4a3a8dee07d7fce62f03feb1cc06bff15598e260af5da8674c80af34de1e1c not found: ID does not exist" Sep 17 23:32:24.963644 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.963621 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh"] Sep 17 23:32:24.970095 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.970076 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b52922-64b76f5f58-js8gh"] Sep 17 23:32:24.979987 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.979968 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv"] Sep 17 23:32:24.984606 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:24.984583 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b52922-5cdf4b5945-74njv"] Sep 17 23:32:26.287306 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:26.287273 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d8d906-d030-4731-94ac-3522ad0b41bb" path="/var/lib/kubelet/pods/45d8d906-d030-4731-94ac-3522ad0b41bb/volumes" Sep 17 23:32:26.287670 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:26.287615 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d67396b8-6a45-456f-b541-45defaca6bf6" path="/var/lib/kubelet/pods/d67396b8-6a45-456f-b541-45defaca6bf6/volumes" Sep 17 23:32:30.509196 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:30.509163 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd"] Sep 17 23:32:30.509562 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:30.509380 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" containerID="cri-o://a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083" gracePeriod=30 Sep 17 23:32:33.393571 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.393534 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb"] Sep 17 23:32:33.394255 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394230 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2283078-2189-4a3b-a4f0-6e0012aeb334" containerName="payload-processing" Sep 17 23:32:33.394255 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394257 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2283078-2189-4a3b-a4f0-6e0012aeb334" containerName="payload-processing" Sep 17 23:32:33.394436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394281 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" Sep 17 23:32:33.394436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394289 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" Sep 17 23:32:33.394436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394302 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db5484f-87d6-4205-a721-19836998c3ed" containerName="istio-proxy" Sep 17 23:32:33.394436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394309 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db5484f-87d6-4205-a721-19836998c3ed" containerName="istio-proxy" Sep 17 23:32:33.394436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394323 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db5484f-87d6-4205-a721-19836998c3ed" containerName="istio-proxy" Sep 17 23:32:33.394436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394330 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db5484f-87d6-4205-a721-19836998c3ed" containerName="istio-proxy" Sep 17 23:32:33.394436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394340 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a36dcca-ae19-40c1-9769-3b8ee9102e56" containerName="maas-api" Sep 17 23:32:33.394436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394346 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a36dcca-ae19-40c1-9769-3b8ee9102e56" containerName="maas-api" Sep 17 23:32:33.394436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394363 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d67396b8-6a45-456f-b541-45defaca6bf6" containerName="payload-pre-processing" Sep 17 23:32:33.394436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394372 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d67396b8-6a45-456f-b541-45defaca6bf6" containerName="payload-pre-processing" Sep 17 23:32:33.394436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394389 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d8d906-d030-4731-94ac-3522ad0b41bb" containerName="payload-processing" Sep 17 23:32:33.394436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394397 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d8d906-d030-4731-94ac-3522ad0b41bb" containerName="payload-processing" Sep 17 23:32:33.395059 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394495 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" Sep 17 23:32:33.395059 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394507 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d8d906-d030-4731-94ac-3522ad0b41bb" containerName="payload-processing" Sep 17 23:32:33.395059 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394519 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db5484f-87d6-4205-a721-19836998c3ed" containerName="istio-proxy" Sep 17 23:32:33.395059 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394528 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a36dcca-ae19-40c1-9769-3b8ee9102e56" containerName="maas-api" Sep 17 23:32:33.395059 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394535 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2283078-2189-4a3b-a4f0-6e0012aeb334" containerName="payload-processing" Sep 17 23:32:33.395059 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394545 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db5484f-87d6-4205-a721-19836998c3ed" containerName="istio-proxy" Sep 17 23:32:33.395059 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394556 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d67396b8-6a45-456f-b541-45defaca6bf6" containerName="payload-pre-processing" Sep 17 23:32:33.395059 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.394566 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" Sep 17 23:32:33.400028 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.400008 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb" Sep 17 23:32:33.403036 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.403010 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d9szq\"" Sep 17 23:32:33.405774 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.405752 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb"] Sep 17 23:32:33.504020 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.503990 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6704a5ff-1b3b-4f8d-a6cf-6c722f471a13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb\" (UID: \"6704a5ff-1b3b-4f8d-a6cf-6c722f471a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb" Sep 17 23:32:33.604952 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.604885 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6704a5ff-1b3b-4f8d-a6cf-6c722f471a13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb\" (UID: \"6704a5ff-1b3b-4f8d-a6cf-6c722f471a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb" Sep 17 23:32:33.605455 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.605436 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6704a5ff-1b3b-4f8d-a6cf-6c722f471a13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb\" (UID: \"6704a5ff-1b3b-4f8d-a6cf-6c722f471a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb" Sep 17 23:32:33.713595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.713569 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb" Sep 17 23:32:33.985178 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:33.985139 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb"] Sep 17 23:32:33.986658 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:32:33.986627 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6704a5ff_1b3b_4f8d_a6cf_6c722f471a13.slice/crio-5c0f65b3d8a8b8ebbaf178e83991738dfe7440315a81b6630b07ac5cf69f849b WatchSource:0}: Error finding container 5c0f65b3d8a8b8ebbaf178e83991738dfe7440315a81b6630b07ac5cf69f849b: Status 404 returned error can't find the container with id 5c0f65b3d8a8b8ebbaf178e83991738dfe7440315a81b6630b07ac5cf69f849b Sep 17 23:32:34.364571 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:34.364530 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 23:32:34.364783 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:34.364593 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 23:32:34.995184 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:34.995098 2559 generic.go:356] "Generic (PLEG): container finished" podID="6704a5ff-1b3b-4f8d-a6cf-6c722f471a13" containerID="c7562320ae8c67b950197b7b23eb0931c846e2cc3ab1a455ed9046a3184f7daf" exitCode=0 Sep 17 23:32:34.995521 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:34.995175 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb" event={"ID":"6704a5ff-1b3b-4f8d-a6cf-6c722f471a13","Type":"ContainerDied","Data":"c7562320ae8c67b950197b7b23eb0931c846e2cc3ab1a455ed9046a3184f7daf"} Sep 17 23:32:34.995521 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:34.995229 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb" event={"ID":"6704a5ff-1b3b-4f8d-a6cf-6c722f471a13","Type":"ContainerStarted","Data":"5c0f65b3d8a8b8ebbaf178e83991738dfe7440315a81b6630b07ac5cf69f849b"} Sep 17 23:32:35.780500 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.780433 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:32:35.928190 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928145 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2xqs\" (UniqueName: \"kubernetes.io/projected/59837b90-41e0-4b62-a77f-5b100840f529-kube-api-access-g2xqs\") pod \"59837b90-41e0-4b62-a77f-5b100840f529\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " Sep 17 23:32:35.928190 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928195 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59837b90-41e0-4b62-a77f-5b100840f529-istio-token\") pod \"59837b90-41e0-4b62-a77f-5b100840f529\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " Sep 17 23:32:35.928433 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928219 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-workload-socket\") pod \"59837b90-41e0-4b62-a77f-5b100840f529\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " Sep 17 23:32:35.928433 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928251 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-istio-envoy\") pod \"59837b90-41e0-4b62-a77f-5b100840f529\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " Sep 17 23:32:35.928433 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928278 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-credential-socket\") pod \"59837b90-41e0-4b62-a77f-5b100840f529\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " Sep 17 23:32:35.928433 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928331 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-istio-data\") pod \"59837b90-41e0-4b62-a77f-5b100840f529\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " Sep 17 23:32:35.928433 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928379 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59837b90-41e0-4b62-a77f-5b100840f529-istiod-ca-cert\") pod \"59837b90-41e0-4b62-a77f-5b100840f529\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " Sep 17 23:32:35.928433 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928432 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-workload-certs\") pod \"59837b90-41e0-4b62-a77f-5b100840f529\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " Sep 17 23:32:35.928817 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928479 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "59837b90-41e0-4b62-a77f-5b100840f529" (UID: "59837b90-41e0-4b62-a77f-5b100840f529"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:35.928817 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928491 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59837b90-41e0-4b62-a77f-5b100840f529-istio-podinfo\") pod \"59837b90-41e0-4b62-a77f-5b100840f529\" (UID: \"59837b90-41e0-4b62-a77f-5b100840f529\") " Sep 17 23:32:35.928817 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928691 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "59837b90-41e0-4b62-a77f-5b100840f529" (UID: "59837b90-41e0-4b62-a77f-5b100840f529"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:35.928817 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928693 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-istio-data" (OuterVolumeSpecName: "istio-data") pod "59837b90-41e0-4b62-a77f-5b100840f529" (UID: "59837b90-41e0-4b62-a77f-5b100840f529"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:35.929055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928810 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59837b90-41e0-4b62-a77f-5b100840f529-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "59837b90-41e0-4b62-a77f-5b100840f529" (UID: "59837b90-41e0-4b62-a77f-5b100840f529"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:32:35.929055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928874 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:35.929055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928892 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:35.929055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928930 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:35.929055 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.928982 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "59837b90-41e0-4b62-a77f-5b100840f529" (UID: "59837b90-41e0-4b62-a77f-5b100840f529"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:35.930792 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.930741 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59837b90-41e0-4b62-a77f-5b100840f529-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "59837b90-41e0-4b62-a77f-5b100840f529" (UID: "59837b90-41e0-4b62-a77f-5b100840f529"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:32:35.930792 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.930762 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59837b90-41e0-4b62-a77f-5b100840f529-kube-api-access-g2xqs" (OuterVolumeSpecName: "kube-api-access-g2xqs") pod "59837b90-41e0-4b62-a77f-5b100840f529" (UID: "59837b90-41e0-4b62-a77f-5b100840f529"). InnerVolumeSpecName "kube-api-access-g2xqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:35.930974 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.930845 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "59837b90-41e0-4b62-a77f-5b100840f529" (UID: "59837b90-41e0-4b62-a77f-5b100840f529"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:35.931064 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:35.931043 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59837b90-41e0-4b62-a77f-5b100840f529-istio-token" (OuterVolumeSpecName: "istio-token") pod "59837b90-41e0-4b62-a77f-5b100840f529" (UID: "59837b90-41e0-4b62-a77f-5b100840f529"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:36.002778 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.002744 2559 generic.go:356] "Generic (PLEG): container finished" podID="59837b90-41e0-4b62-a77f-5b100840f529" containerID="a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083" exitCode=0 Sep 17 23:32:36.003189 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.002801 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" Sep 17 23:32:36.003189 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.002836 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" event={"ID":"59837b90-41e0-4b62-a77f-5b100840f529","Type":"ContainerDied","Data":"a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083"} Sep 17 23:32:36.003189 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.002934 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd" event={"ID":"59837b90-41e0-4b62-a77f-5b100840f529","Type":"ContainerDied","Data":"76c7470d2b74be5bd52c4ecb73eea9d3d663a5711e725dfa5b6538369bf66133"} Sep 17 23:32:36.003189 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.002957 2559 scope.go:117] "RemoveContainer" containerID="a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083" Sep 17 23:32:36.016487 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.016451 2559 scope.go:117] "RemoveContainer" containerID="0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247" Sep 17 23:32:36.027533 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.027480 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd"] Sep 17 23:32:36.027607 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.027552 2559 scope.go:117] "RemoveContainer" containerID="a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083" Sep 17 23:32:36.027879 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:32:36.027855 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083\": container with ID starting with a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083 not found: ID does not exist" containerID="a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083" Sep 17 23:32:36.027944 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.027887 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083"} err="failed to get container status \"a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083\": rpc error: code = NotFound desc = could not find container \"a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083\": container with ID starting with a4ddec63f35034b40d95f3fc299c78d4ff6efbd4f1994f5e14d605e6ca659083 not found: ID does not exist" Sep 17 23:32:36.027944 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.027932 2559 scope.go:117] "RemoveContainer" containerID="0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247" Sep 17 23:32:36.028299 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:32:36.028267 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247\": container with ID starting with 0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247 not found: ID does not exist" containerID="0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247" Sep 17 23:32:36.028421 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.028302 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247"} err="failed to get container status \"0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247\": rpc error: code = NotFound desc = could not find container \"0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247\": container with ID starting with 0a9d5b2efa4d441d92e8e42db9cba205669e261d738dc175d62ce96c04c81247 not found: ID does not exist" Sep 17 23:32:36.029975 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.029941 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59837b90-41e0-4b62-a77f-5b100840f529-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:36.029975 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.029975 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2xqs\" (UniqueName: \"kubernetes.io/projected/59837b90-41e0-4b62-a77f-5b100840f529-kube-api-access-g2xqs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:36.029975 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.029984 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59837b90-41e0-4b62-a77f-5b100840f529-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:36.030143 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.029993 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:36.030143 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.030001 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59837b90-41e0-4b62-a77f-5b100840f529-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:36.030143 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.030008 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59837b90-41e0-4b62-a77f-5b100840f529-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:36.031243 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.031191 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b52922-openshift-default-76f7cc4666-b2cvd"] Sep 17 23:32:36.141393 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.141368 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb" Sep 17 23:32:36.232111 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.232079 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6704a5ff-1b3b-4f8d-a6cf-6c722f471a13-maas-api-service-ca\") pod \"6704a5ff-1b3b-4f8d-a6cf-6c722f471a13\" (UID: \"6704a5ff-1b3b-4f8d-a6cf-6c722f471a13\") " Sep 17 23:32:36.232407 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.232386 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6704a5ff-1b3b-4f8d-a6cf-6c722f471a13-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6704a5ff-1b3b-4f8d-a6cf-6c722f471a13" (UID: "6704a5ff-1b3b-4f8d-a6cf-6c722f471a13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:32:36.288724 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.288648 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59837b90-41e0-4b62-a77f-5b100840f529" path="/var/lib/kubelet/pods/59837b90-41e0-4b62-a77f-5b100840f529/volumes" Sep 17 23:32:36.332954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:36.332929 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6704a5ff-1b3b-4f8d-a6cf-6c722f471a13-maas-api-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:37.011487 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:37.011461 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb" Sep 17 23:32:37.011846 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:37.011505 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb" event={"ID":"6704a5ff-1b3b-4f8d-a6cf-6c722f471a13","Type":"ContainerDied","Data":"5c0f65b3d8a8b8ebbaf178e83991738dfe7440315a81b6630b07ac5cf69f849b"} Sep 17 23:32:37.011846 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:37.011549 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c0f65b3d8a8b8ebbaf178e83991738dfe7440315a81b6630b07ac5cf69f849b" Sep 17 23:32:41.793677 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:41.793643 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm"] Sep 17 23:32:41.794447 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:41.794391 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" podUID="1268244f-5d3c-45b4-911a-621268865ccf" containerName="maas-api" containerID="cri-o://f5c2c43ea075ba8d51d87ffcf5b0b45c7e5fa477ac4c1033d213d1fa946bcb90" gracePeriod=30 Sep 17 23:32:41.823747 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:41.823723 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k"] Sep 17 23:32:41.827118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:41.827094 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-efffde-29828130-qhq5k"] Sep 17 23:32:41.859192 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:41.859166 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7"] Sep 17 23:32:41.859389 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:41.859366 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" podUID="0356abbb-3e8e-41fb-bace-a1375b4cd869" containerName="payload-pre-processing" containerID="cri-o://ef333522a2752632a4832ad3f230ac1731641bdf049d78997904712da81f3d2b" gracePeriod=30 Sep 17 23:32:42.044840 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.042242 2559 generic.go:356] "Generic (PLEG): container finished" podID="1268244f-5d3c-45b4-911a-621268865ccf" containerID="f5c2c43ea075ba8d51d87ffcf5b0b45c7e5fa477ac4c1033d213d1fa946bcb90" exitCode=0 Sep 17 23:32:42.044840 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.042367 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" event={"ID":"1268244f-5d3c-45b4-911a-621268865ccf","Type":"ContainerDied","Data":"f5c2c43ea075ba8d51d87ffcf5b0b45c7e5fa477ac4c1033d213d1fa946bcb90"} Sep 17 23:32:42.044840 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.044602 2559 generic.go:356] "Generic (PLEG): container finished" podID="0356abbb-3e8e-41fb-bace-a1375b4cd869" containerID="ef333522a2752632a4832ad3f230ac1731641bdf049d78997904712da81f3d2b" exitCode=0 Sep 17 23:32:42.044840 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.044754 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" event={"ID":"0356abbb-3e8e-41fb-bace-a1375b4cd869","Type":"ContainerDied","Data":"ef333522a2752632a4832ad3f230ac1731641bdf049d78997904712da81f3d2b"} Sep 17 23:32:42.119408 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.119372 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:32:42.157453 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.157430 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" Sep 17 23:32:42.288254 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.288228 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0356abbb-3e8e-41fb-bace-a1375b4cd869-plugins-config-volume\") pod \"0356abbb-3e8e-41fb-bace-a1375b4cd869\" (UID: \"0356abbb-3e8e-41fb-bace-a1375b4cd869\") " Sep 17 23:32:42.288432 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.288284 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-metrics-certs\") pod \"1268244f-5d3c-45b4-911a-621268865ccf\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " Sep 17 23:32:42.288432 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.288316 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b77n\" (UniqueName: \"kubernetes.io/projected/0356abbb-3e8e-41fb-bace-a1375b4cd869-kube-api-access-4b77n\") pod \"0356abbb-3e8e-41fb-bace-a1375b4cd869\" (UID: \"0356abbb-3e8e-41fb-bace-a1375b4cd869\") " Sep 17 23:32:42.288432 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.288336 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1268244f-5d3c-45b4-911a-621268865ccf-openshift-service-ca\") pod \"1268244f-5d3c-45b4-911a-621268865ccf\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " Sep 17 23:32:42.288432 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.288372 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tw5f\" (UniqueName: \"kubernetes.io/projected/1268244f-5d3c-45b4-911a-621268865ccf-kube-api-access-9tw5f\") pod \"1268244f-5d3c-45b4-911a-621268865ccf\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " Sep 17 23:32:42.288432 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.288398 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-maas-api-tls\") pod \"1268244f-5d3c-45b4-911a-621268865ccf\" (UID: \"1268244f-5d3c-45b4-911a-621268865ccf\") " Sep 17 23:32:42.288717 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.288619 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0356abbb-3e8e-41fb-bace-a1375b4cd869-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0356abbb-3e8e-41fb-bace-a1375b4cd869" (UID: "0356abbb-3e8e-41fb-bace-a1375b4cd869"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:32:42.288717 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.288705 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1268244f-5d3c-45b4-911a-621268865ccf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1268244f-5d3c-45b4-911a-621268865ccf" (UID: "1268244f-5d3c-45b4-911a-621268865ccf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:32:42.288852 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.288769 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0356abbb-3e8e-41fb-bace-a1375b4cd869-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:42.288852 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.288771 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db6f3ef4-560d-405d-afa3-95194bf94640" path="/var/lib/kubelet/pods/db6f3ef4-560d-405d-afa3-95194bf94640/volumes" Sep 17 23:32:42.288852 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.288782 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1268244f-5d3c-45b4-911a-621268865ccf-openshift-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:42.290671 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.290643 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1268244f-5d3c-45b4-911a-621268865ccf-kube-api-access-9tw5f" (OuterVolumeSpecName: "kube-api-access-9tw5f") pod "1268244f-5d3c-45b4-911a-621268865ccf" (UID: "1268244f-5d3c-45b4-911a-621268865ccf"). InnerVolumeSpecName "kube-api-access-9tw5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:42.290790 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.290650 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0356abbb-3e8e-41fb-bace-a1375b4cd869-kube-api-access-4b77n" (OuterVolumeSpecName: "kube-api-access-4b77n") pod "0356abbb-3e8e-41fb-bace-a1375b4cd869" (UID: "0356abbb-3e8e-41fb-bace-a1375b4cd869"). InnerVolumeSpecName "kube-api-access-4b77n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:42.290928 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.290885 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1268244f-5d3c-45b4-911a-621268865ccf" (UID: "1268244f-5d3c-45b4-911a-621268865ccf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:32:42.291032 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.290959 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1268244f-5d3c-45b4-911a-621268865ccf" (UID: "1268244f-5d3c-45b4-911a-621268865ccf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:32:42.389974 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.389889 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b77n\" (UniqueName: \"kubernetes.io/projected/0356abbb-3e8e-41fb-bace-a1375b4cd869-kube-api-access-4b77n\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:42.389974 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.389937 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tw5f\" (UniqueName: \"kubernetes.io/projected/1268244f-5d3c-45b4-911a-621268865ccf-kube-api-access-9tw5f\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:42.389974 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.389948 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-maas-api-tls\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:42.389974 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:42.389956 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1268244f-5d3c-45b4-911a-621268865ccf-metrics-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:43.052034 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:43.052005 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" Sep 17 23:32:43.052461 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:43.052039 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7" event={"ID":"0356abbb-3e8e-41fb-bace-a1375b4cd869","Type":"ContainerDied","Data":"a3f841d53133f0f7637428205f9b4cb2539bd04f7574a988af77529f7de932f3"} Sep 17 23:32:43.052461 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:43.052101 2559 scope.go:117] "RemoveContainer" containerID="ef333522a2752632a4832ad3f230ac1731641bdf049d78997904712da81f3d2b" Sep 17 23:32:43.053576 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:43.053546 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" event={"ID":"1268244f-5d3c-45b4-911a-621268865ccf","Type":"ContainerDied","Data":"86ff67f25e1b1f561a4d4ef3fdc622e7f33e17e24f50f9d104e87b285d6168b8"} Sep 17 23:32:43.053679 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:43.053601 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm" Sep 17 23:32:43.062526 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:43.062509 2559 scope.go:117] "RemoveContainer" containerID="f5c2c43ea075ba8d51d87ffcf5b0b45c7e5fa477ac4c1033d213d1fa946bcb90" Sep 17 23:32:43.074315 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:43.074294 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7"] Sep 17 23:32:43.077350 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:43.077330 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-efffde-79c576d8f8-s2tr7"] Sep 17 23:32:43.087385 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:43.087363 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm"] Sep 17 23:32:43.092781 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:43.092762 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-efffde-557699699b-9mrfm"] Sep 17 23:32:44.287677 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.287643 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0356abbb-3e8e-41fb-bace-a1375b4cd869" path="/var/lib/kubelet/pods/0356abbb-3e8e-41fb-bace-a1375b4cd869/volumes" Sep 17 23:32:44.288058 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.288043 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1268244f-5d3c-45b4-911a-621268865ccf" path="/var/lib/kubelet/pods/1268244f-5d3c-45b4-911a-621268865ccf/volumes" Sep 17 23:32:44.508544 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.508513 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc"] Sep 17 23:32:44.508757 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.508728 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" podUID="e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9" containerName="payload-processing" containerID="cri-o://30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b" gracePeriod=30 Sep 17 23:32:44.519500 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.519470 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk"] Sep 17 23:32:44.519745 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.519700 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" podUID="cc2e17bb-783b-49ee-9104-f4fde4ce610c" containerName="payload-pre-processing" containerID="cri-o://7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d" gracePeriod=30 Sep 17 23:32:44.810666 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.810641 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" Sep 17 23:32:44.824802 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.824783 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" Sep 17 23:32:44.914506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.914473 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pp24\" (UniqueName: \"kubernetes.io/projected/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9-kube-api-access-7pp24\") pod \"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9\" (UID: \"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9\") " Sep 17 23:32:44.914506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.914510 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd2lg\" (UniqueName: \"kubernetes.io/projected/cc2e17bb-783b-49ee-9104-f4fde4ce610c-kube-api-access-xd2lg\") pod \"cc2e17bb-783b-49ee-9104-f4fde4ce610c\" (UID: \"cc2e17bb-783b-49ee-9104-f4fde4ce610c\") " Sep 17 23:32:44.914718 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.914566 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2e17bb-783b-49ee-9104-f4fde4ce610c-plugins-config-volume\") pod \"cc2e17bb-783b-49ee-9104-f4fde4ce610c\" (UID: \"cc2e17bb-783b-49ee-9104-f4fde4ce610c\") " Sep 17 23:32:44.914718 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.914613 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9-plugins-config-volume\") pod \"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9\" (UID: \"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9\") " Sep 17 23:32:44.915057 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.915015 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9" (UID: "e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:32:44.915057 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.915018 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2e17bb-783b-49ee-9104-f4fde4ce610c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc2e17bb-783b-49ee-9104-f4fde4ce610c" (UID: "cc2e17bb-783b-49ee-9104-f4fde4ce610c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:32:44.916954 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.916928 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2e17bb-783b-49ee-9104-f4fde4ce610c-kube-api-access-xd2lg" (OuterVolumeSpecName: "kube-api-access-xd2lg") pod "cc2e17bb-783b-49ee-9104-f4fde4ce610c" (UID: "cc2e17bb-783b-49ee-9104-f4fde4ce610c"). InnerVolumeSpecName "kube-api-access-xd2lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:44.917087 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:44.917019 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9-kube-api-access-7pp24" (OuterVolumeSpecName: "kube-api-access-7pp24") pod "e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9" (UID: "e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9"). InnerVolumeSpecName "kube-api-access-7pp24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:45.016158 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.016129 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2e17bb-783b-49ee-9104-f4fde4ce610c-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:45.016158 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.016153 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:45.016158 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.016164 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pp24\" (UniqueName: \"kubernetes.io/projected/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9-kube-api-access-7pp24\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:45.016360 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.016173 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd2lg\" (UniqueName: \"kubernetes.io/projected/cc2e17bb-783b-49ee-9104-f4fde4ce610c-kube-api-access-xd2lg\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:45.068289 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.068216 2559 generic.go:356] "Generic (PLEG): container finished" podID="e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9" containerID="30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b" exitCode=0 Sep 17 23:32:45.068289 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.068254 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" Sep 17 23:32:45.068467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.068303 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" event={"ID":"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9","Type":"ContainerDied","Data":"30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b"} Sep 17 23:32:45.068467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.068352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc" event={"ID":"e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9","Type":"ContainerDied","Data":"e83ade0937dcb50d9628bed522f18bef7ebf35c79407691625bcb3a926190171"} Sep 17 23:32:45.068467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.068372 2559 scope.go:117] "RemoveContainer" containerID="30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b" Sep 17 23:32:45.069723 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.069700 2559 generic.go:356] "Generic (PLEG): container finished" podID="cc2e17bb-783b-49ee-9104-f4fde4ce610c" containerID="7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d" exitCode=0 Sep 17 23:32:45.069838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.069728 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" event={"ID":"cc2e17bb-783b-49ee-9104-f4fde4ce610c","Type":"ContainerDied","Data":"7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d"} Sep 17 23:32:45.069838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.069743 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" Sep 17 23:32:45.069838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.069777 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk" event={"ID":"cc2e17bb-783b-49ee-9104-f4fde4ce610c","Type":"ContainerDied","Data":"69a1125108f4579725574b06d5071b0ad752e91691d97d62cfc9db9425e36639"} Sep 17 23:32:45.078426 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.078406 2559 scope.go:117] "RemoveContainer" containerID="30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b" Sep 17 23:32:45.078705 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:32:45.078681 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b\": container with ID starting with 30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b not found: ID does not exist" containerID="30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b" Sep 17 23:32:45.078782 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.078714 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b"} err="failed to get container status \"30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b\": rpc error: code = NotFound desc = could not find container \"30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b\": container with ID starting with 30672d49929033d6340995940567f7dcd1527ebe51c961bde122487fd118429b not found: ID does not exist" Sep 17 23:32:45.078782 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.078736 2559 scope.go:117] "RemoveContainer" containerID="7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d" Sep 17 23:32:45.088671 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.088656 2559 scope.go:117] "RemoveContainer" containerID="7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d" Sep 17 23:32:45.089142 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:32:45.089123 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d\": container with ID starting with 7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d not found: ID does not exist" containerID="7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d" Sep 17 23:32:45.089213 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.089152 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d"} err="failed to get container status \"7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d\": rpc error: code = NotFound desc = could not find container \"7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d\": container with ID starting with 7b90c0faec80b85ad766f48b80f6727cbf3780390b8cec5ca7ad963e9f18e39d not found: ID does not exist" Sep 17 23:32:45.119941 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.119918 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk"] Sep 17 23:32:45.133219 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.133197 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-825f30-55bbd6c5fd-bh7rk"] Sep 17 23:32:45.154374 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.154354 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc"] Sep 17 23:32:45.162681 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.162659 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-825f30-85d7bd455-jjvvc"] Sep 17 23:32:45.699923 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:45.699880 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:32:46.287088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:46.287057 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc2e17bb-783b-49ee-9104-f4fde4ce610c" path="/var/lib/kubelet/pods/cc2e17bb-783b-49ee-9104-f4fde4ce610c/volumes" Sep 17 23:32:46.287445 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:46.287432 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9" path="/var/lib/kubelet/pods/e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9/volumes" Sep 17 23:32:48.723444 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:48.723412 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc"] Sep 17 23:32:48.723888 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:48.723611 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerName="istio-proxy" containerID="cri-o://844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd" gracePeriod=30 Sep 17 23:32:50.653527 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:50.653494 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7"] Sep 17 23:32:50.655261 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:50.655238 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-736809-29828130-fb7t7"] Sep 17 23:32:50.853088 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:50.853009 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb"] Sep 17 23:32:50.859101 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:50.859074 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-736809-vqzjb"] Sep 17 23:32:52.287196 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:52.287164 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6704a5ff-1b3b-4f8d-a6cf-6c722f471a13" path="/var/lib/kubelet/pods/6704a5ff-1b3b-4f8d-a6cf-6c722f471a13/volumes" Sep 17 23:32:52.287576 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:52.287454 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba4275bb-fb03-49c9-9199-53601069a9a9" path="/var/lib/kubelet/pods/ba4275bb-fb03-49c9-9199-53601069a9a9/volumes" Sep 17 23:32:53.981958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.981936 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:32:53.994685 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.994631 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-workload-certs\") pod \"6bba8ad1-648f-42ec-a7de-bb2673058f29\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " Sep 17 23:32:53.994685 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.994670 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-envoy\") pod \"6bba8ad1-648f-42ec-a7de-bb2673058f29\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " Sep 17 23:32:53.994866 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.994686 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-credential-socket\") pod \"6bba8ad1-648f-42ec-a7de-bb2673058f29\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " Sep 17 23:32:53.994866 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.994717 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-workload-socket\") pod \"6bba8ad1-648f-42ec-a7de-bb2673058f29\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " Sep 17 23:32:53.994866 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.994736 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-token\") pod \"6bba8ad1-648f-42ec-a7de-bb2673058f29\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " Sep 17 23:32:53.994866 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.994755 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bba8ad1-648f-42ec-a7de-bb2673058f29-istiod-ca-cert\") pod \"6bba8ad1-648f-42ec-a7de-bb2673058f29\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " Sep 17 23:32:53.994866 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.994776 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8b9p\" (UniqueName: \"kubernetes.io/projected/6bba8ad1-648f-42ec-a7de-bb2673058f29-kube-api-access-k8b9p\") pod \"6bba8ad1-648f-42ec-a7de-bb2673058f29\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " Sep 17 23:32:53.994866 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.994850 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-data\") pod \"6bba8ad1-648f-42ec-a7de-bb2673058f29\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " Sep 17 23:32:53.995197 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.994872 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-podinfo\") pod \"6bba8ad1-648f-42ec-a7de-bb2673058f29\" (UID: \"6bba8ad1-648f-42ec-a7de-bb2673058f29\") " Sep 17 23:32:53.995197 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.994948 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6bba8ad1-648f-42ec-a7de-bb2673058f29" (UID: "6bba8ad1-648f-42ec-a7de-bb2673058f29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:53.995197 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.994983 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6bba8ad1-648f-42ec-a7de-bb2673058f29" (UID: "6bba8ad1-648f-42ec-a7de-bb2673058f29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:53.995197 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.995096 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6bba8ad1-648f-42ec-a7de-bb2673058f29" (UID: "6bba8ad1-648f-42ec-a7de-bb2673058f29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:53.995392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.995259 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-data" (OuterVolumeSpecName: "istio-data") pod "6bba8ad1-648f-42ec-a7de-bb2673058f29" (UID: "6bba8ad1-648f-42ec-a7de-bb2673058f29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:53.995392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.995284 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:53.995392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.995299 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:53.995392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.995308 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:53.995603 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.995421 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bba8ad1-648f-42ec-a7de-bb2673058f29-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6bba8ad1-648f-42ec-a7de-bb2673058f29" (UID: "6bba8ad1-648f-42ec-a7de-bb2673058f29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:32:53.996888 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.996858 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6bba8ad1-648f-42ec-a7de-bb2673058f29" (UID: "6bba8ad1-648f-42ec-a7de-bb2673058f29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:32:53.997022 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.996889 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-token" (OuterVolumeSpecName: "istio-token") pod "6bba8ad1-648f-42ec-a7de-bb2673058f29" (UID: "6bba8ad1-648f-42ec-a7de-bb2673058f29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:53.997022 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.996940 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6bba8ad1-648f-42ec-a7de-bb2673058f29" (UID: "6bba8ad1-648f-42ec-a7de-bb2673058f29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:32:53.997109 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:53.997030 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bba8ad1-648f-42ec-a7de-bb2673058f29-kube-api-access-k8b9p" (OuterVolumeSpecName: "kube-api-access-k8b9p") pod "6bba8ad1-648f-42ec-a7de-bb2673058f29" (UID: "6bba8ad1-648f-42ec-a7de-bb2673058f29"). InnerVolumeSpecName "kube-api-access-k8b9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:32:54.095805 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.095775 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:54.095805 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.095800 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:54.095805 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.095809 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:54.096047 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.095817 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bba8ad1-648f-42ec-a7de-bb2673058f29-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:54.096047 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.095825 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bba8ad1-648f-42ec-a7de-bb2673058f29-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:54.096047 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.095833 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8b9p\" (UniqueName: \"kubernetes.io/projected/6bba8ad1-648f-42ec-a7de-bb2673058f29-kube-api-access-k8b9p\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:32:54.123992 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.123958 2559 generic.go:356] "Generic (PLEG): container finished" podID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerID="844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd" exitCode=0 Sep 17 23:32:54.124131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.124044 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" event={"ID":"6bba8ad1-648f-42ec-a7de-bb2673058f29","Type":"ContainerDied","Data":"844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd"} Sep 17 23:32:54.127936 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.124486 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" Sep 17 23:32:54.127936 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.124527 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc" event={"ID":"6bba8ad1-648f-42ec-a7de-bb2673058f29","Type":"ContainerDied","Data":"910f4ce5f9259c14bf91e81053539bfb02075d2fb3c0e85ed74ef8f7df0e336f"} Sep 17 23:32:54.127936 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.124565 2559 scope.go:117] "RemoveContainer" containerID="844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd" Sep 17 23:32:54.141948 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.141927 2559 scope.go:117] "RemoveContainer" containerID="2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2" Sep 17 23:32:54.150566 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.150545 2559 scope.go:117] "RemoveContainer" containerID="844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd" Sep 17 23:32:54.150863 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:32:54.150844 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd\": container with ID starting with 844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd not found: ID does not exist" containerID="844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd" Sep 17 23:32:54.151000 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.150873 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd"} err="failed to get container status \"844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd\": rpc error: code = NotFound desc = could not find container \"844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd\": container with ID starting with 844569f629f16aed48a1072886f84be11774244df8b40eddd0904ea65cd952fd not found: ID does not exist" Sep 17 23:32:54.151000 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.150890 2559 scope.go:117] "RemoveContainer" containerID="2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2" Sep 17 23:32:54.151196 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:32:54.151178 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2\": container with ID starting with 2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2 not found: ID does not exist" containerID="2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2" Sep 17 23:32:54.151266 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.151201 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2"} err="failed to get container status \"2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2\": rpc error: code = NotFound desc = could not find container \"2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2\": container with ID starting with 2678af2671da35357b102611f2b65cdbb3b0c156425a746c7f24c5bf71d6d1e2 not found: ID does not exist" Sep 17 23:32:54.152640 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.152619 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc"] Sep 17 23:32:54.155590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.155568 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-efffde-openshift-default-7bf694c556-t5sjc"] Sep 17 23:32:54.288354 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:32:54.288286 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" path="/var/lib/kubelet/pods/6bba8ad1-648f-42ec-a7de-bb2673058f29/volumes" Sep 17 23:33:10.400294 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:10.400210 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:33:12.303004 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.302968 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7853-fd8981"] Sep 17 23:33:12.303414 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303398 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1268244f-5d3c-45b4-911a-621268865ccf" containerName="maas-api" Sep 17 23:33:12.303414 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303416 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1268244f-5d3c-45b4-911a-621268865ccf" containerName="maas-api" Sep 17 23:33:12.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303426 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerName="istio-proxy" Sep 17 23:33:12.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303432 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerName="istio-proxy" Sep 17 23:33:12.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303445 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2e17bb-783b-49ee-9104-f4fde4ce610c" containerName="payload-pre-processing" Sep 17 23:33:12.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303450 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2e17bb-783b-49ee-9104-f4fde4ce610c" containerName="payload-pre-processing" Sep 17 23:33:12.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303460 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" Sep 17 23:33:12.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303465 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e54ba1-48af-4cdb-a253-9e0f741c85a4" containerName="istio-proxy" Sep 17 23:33:12.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303479 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9" containerName="payload-processing" Sep 17 23:33:12.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303484 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9" containerName="payload-processing" Sep 17 23:33:12.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303490 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerName="istio-proxy" Sep 17 23:33:12.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303494 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerName="istio-proxy" Sep 17 23:33:12.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303500 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" Sep 17 23:33:12.303503 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303504 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303516 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303523 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303536 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0356abbb-3e8e-41fb-bace-a1375b4cd869" containerName="payload-pre-processing" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303544 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0356abbb-3e8e-41fb-bace-a1375b4cd869" containerName="payload-pre-processing" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303552 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6704a5ff-1b3b-4f8d-a6cf-6c722f471a13" containerName="revoke-keys" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303557 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6704a5ff-1b3b-4f8d-a6cf-6c722f471a13" containerName="revoke-keys" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303618 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6704a5ff-1b3b-4f8d-a6cf-6c722f471a13" containerName="revoke-keys" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303624 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303631 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1268244f-5d3c-45b4-911a-621268865ccf" containerName="maas-api" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303636 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2e17bb-783b-49ee-9104-f4fde4ce610c" containerName="payload-pre-processing" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303640 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ae6ed0-cfc8-4bfc-8880-5ef0db85a9a9" containerName="payload-processing" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303646 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0356abbb-3e8e-41fb-bace-a1375b4cd869" containerName="payload-pre-processing" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303652 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerName="istio-proxy" Sep 17 23:33:12.303967 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.303657 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bba8ad1-648f-42ec-a7de-bb2673058f29" containerName="istio-proxy" Sep 17 23:33:12.306780 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.306757 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-fd8981" Sep 17 23:33:12.309178 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.309156 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pml6m\"" Sep 17 23:33:12.313221 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.313199 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-fd8981"] Sep 17 23:33:12.361004 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.360973 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9brp\" (UniqueName: \"kubernetes.io/projected/4614ee7c-4e16-4c49-a8a4-8c442885ebe9-kube-api-access-s9brp\") pod \"test-curl-7853-fd8981\" (UID: \"4614ee7c-4e16-4c49-a8a4-8c442885ebe9\") " pod="opendatahub/test-curl-7853-fd8981" Sep 17 23:33:12.462173 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.462134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9brp\" (UniqueName: \"kubernetes.io/projected/4614ee7c-4e16-4c49-a8a4-8c442885ebe9-kube-api-access-s9brp\") pod \"test-curl-7853-fd8981\" (UID: \"4614ee7c-4e16-4c49-a8a4-8c442885ebe9\") " pod="opendatahub/test-curl-7853-fd8981" Sep 17 23:33:12.469759 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.469724 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9brp\" (UniqueName: \"kubernetes.io/projected/4614ee7c-4e16-4c49-a8a4-8c442885ebe9-kube-api-access-s9brp\") pod \"test-curl-7853-fd8981\" (UID: \"4614ee7c-4e16-4c49-a8a4-8c442885ebe9\") " pod="opendatahub/test-curl-7853-fd8981" Sep 17 23:33:12.617486 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.617402 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-fd8981" Sep 17 23:33:12.889093 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:12.889012 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-fd8981"] Sep 17 23:33:12.892656 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:33:12.892620 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4614ee7c_4e16_4c49_a8a4_8c442885ebe9.slice/crio-f42d1f43d1896c332c4f516b17bbef56361cd0697024f254f522497ec125cf12 WatchSource:0}: Error finding container f42d1f43d1896c332c4f516b17bbef56361cd0697024f254f522497ec125cf12: Status 404 returned error can't find the container with id f42d1f43d1896c332c4f516b17bbef56361cd0697024f254f522497ec125cf12 Sep 17 23:33:13.234384 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:13.234351 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-fd8981" event={"ID":"4614ee7c-4e16-4c49-a8a4-8c442885ebe9","Type":"ContainerStarted","Data":"8b65b21c4c2c96407abf1ce2f8560decf43bf1e921221c1a7b6fd164426dbfe2"} Sep 17 23:33:13.234557 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:13.234406 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-fd8981" event={"ID":"4614ee7c-4e16-4c49-a8a4-8c442885ebe9","Type":"ContainerStarted","Data":"f42d1f43d1896c332c4f516b17bbef56361cd0697024f254f522497ec125cf12"} Sep 17 23:33:13.250408 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:13.250347 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7853-fd8981" podStartSLOduration=1.250327334 podStartE2EDuration="1.250327334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:13.248538854 +0000 UTC m=+1621.366018304" watchObservedRunningTime="2026-09-17 23:33:13.250327334 +0000 UTC m=+1621.367806784" Sep 17 23:33:13.826557 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:13.826516 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7853-fd8981"] Sep 17 23:33:13.828776 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:13.828749 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7853-fd8981"] Sep 17 23:33:14.240721 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:33:14.240683 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7853-fd8981" container="test-curl-7853-fd8981" Sep 17 23:33:15.247518 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:15.247470 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7853-fd8981" podUID="4614ee7c-4e16-4c49-a8a4-8c442885ebe9" containerName="test-curl-7853-fd8981" containerID="cri-o://8b65b21c4c2c96407abf1ce2f8560decf43bf1e921221c1a7b6fd164426dbfe2" gracePeriod=2 Sep 17 23:33:17.433011 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:17.432987 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-fd8981" Sep 17 23:33:17.506395 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:17.506321 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9brp\" (UniqueName: \"kubernetes.io/projected/4614ee7c-4e16-4c49-a8a4-8c442885ebe9-kube-api-access-s9brp\") pod \"4614ee7c-4e16-4c49-a8a4-8c442885ebe9\" (UID: \"4614ee7c-4e16-4c49-a8a4-8c442885ebe9\") " Sep 17 23:33:17.508291 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:17.508269 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4614ee7c-4e16-4c49-a8a4-8c442885ebe9-kube-api-access-s9brp" (OuterVolumeSpecName: "kube-api-access-s9brp") pod "4614ee7c-4e16-4c49-a8a4-8c442885ebe9" (UID: "4614ee7c-4e16-4c49-a8a4-8c442885ebe9"). InnerVolumeSpecName "kube-api-access-s9brp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:33:17.607708 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:17.607671 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9brp\" (UniqueName: \"kubernetes.io/projected/4614ee7c-4e16-4c49-a8a4-8c442885ebe9-kube-api-access-s9brp\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:33:18.266360 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:18.266311 2559 generic.go:356] "Generic (PLEG): container finished" podID="4614ee7c-4e16-4c49-a8a4-8c442885ebe9" containerID="8b65b21c4c2c96407abf1ce2f8560decf43bf1e921221c1a7b6fd164426dbfe2" exitCode=137 Sep 17 23:33:18.266576 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:18.266464 2559 scope.go:117] "RemoveContainer" containerID="8b65b21c4c2c96407abf1ce2f8560decf43bf1e921221c1a7b6fd164426dbfe2" Sep 17 23:33:18.266672 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:18.266656 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-fd8981" Sep 17 23:33:18.278740 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:18.278720 2559 scope.go:117] "RemoveContainer" containerID="8b65b21c4c2c96407abf1ce2f8560decf43bf1e921221c1a7b6fd164426dbfe2" Sep 17 23:33:18.279048 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:33:18.279029 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b65b21c4c2c96407abf1ce2f8560decf43bf1e921221c1a7b6fd164426dbfe2\": container with ID starting with 8b65b21c4c2c96407abf1ce2f8560decf43bf1e921221c1a7b6fd164426dbfe2 not found: ID does not exist" containerID="8b65b21c4c2c96407abf1ce2f8560decf43bf1e921221c1a7b6fd164426dbfe2" Sep 17 23:33:18.279176 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:18.279060 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b65b21c4c2c96407abf1ce2f8560decf43bf1e921221c1a7b6fd164426dbfe2"} err="failed to get container status \"8b65b21c4c2c96407abf1ce2f8560decf43bf1e921221c1a7b6fd164426dbfe2\": rpc error: code = NotFound desc = could not find container \"8b65b21c4c2c96407abf1ce2f8560decf43bf1e921221c1a7b6fd164426dbfe2\": container with ID starting with 8b65b21c4c2c96407abf1ce2f8560decf43bf1e921221c1a7b6fd164426dbfe2 not found: ID does not exist" Sep 17 23:33:18.279268 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:33:18.279241 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7853-fd8981\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="4614ee7c-4e16-4c49-a8a4-8c442885ebe9" pod="opendatahub/test-curl-7853-fd8981" Sep 17 23:33:18.286641 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:18.286615 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4614ee7c-4e16-4c49-a8a4-8c442885ebe9" path="/var/lib/kubelet/pods/4614ee7c-4e16-4c49-a8a4-8c442885ebe9/volumes" Sep 17 23:33:24.113810 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:24.113776 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:33:46.494443 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:33:46.494406 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:34:17.404285 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:17.404240 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:34:30.499138 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:30.499050 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:34:40.993358 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:40.993326 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7853-a6a3d6"] Sep 17 23:34:40.993797 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:40.993716 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4614ee7c-4e16-4c49-a8a4-8c442885ebe9" containerName="test-curl-7853-fd8981" Sep 17 23:34:40.993797 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:40.993728 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4614ee7c-4e16-4c49-a8a4-8c442885ebe9" containerName="test-curl-7853-fd8981" Sep 17 23:34:40.993868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:40.993817 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4614ee7c-4e16-4c49-a8a4-8c442885ebe9" containerName="test-curl-7853-fd8981" Sep 17 23:34:40.993868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:40.993833 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="59837b90-41e0-4b62-a77f-5b100840f529" containerName="istio-proxy" Sep 17 23:34:40.996996 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:40.996971 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-a6a3d6" Sep 17 23:34:40.999518 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:40.999495 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pml6m\"" Sep 17 23:34:41.001109 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:41.001085 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-a6a3d6"] Sep 17 23:34:41.039115 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:41.039078 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk7lx\" (UniqueName: \"kubernetes.io/projected/1d08b131-79b5-4235-90f6-44b10f68a36f-kube-api-access-tk7lx\") pod \"test-curl-7853-a6a3d6\" (UID: \"1d08b131-79b5-4235-90f6-44b10f68a36f\") " pod="opendatahub/test-curl-7853-a6a3d6" Sep 17 23:34:41.139879 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:41.139844 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk7lx\" (UniqueName: \"kubernetes.io/projected/1d08b131-79b5-4235-90f6-44b10f68a36f-kube-api-access-tk7lx\") pod \"test-curl-7853-a6a3d6\" (UID: \"1d08b131-79b5-4235-90f6-44b10f68a36f\") " pod="opendatahub/test-curl-7853-a6a3d6" Sep 17 23:34:41.148660 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:41.148634 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk7lx\" (UniqueName: \"kubernetes.io/projected/1d08b131-79b5-4235-90f6-44b10f68a36f-kube-api-access-tk7lx\") pod \"test-curl-7853-a6a3d6\" (UID: \"1d08b131-79b5-4235-90f6-44b10f68a36f\") " pod="opendatahub/test-curl-7853-a6a3d6" Sep 17 23:34:41.308783 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:41.308703 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-a6a3d6" Sep 17 23:34:41.577078 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:41.577053 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-a6a3d6"] Sep 17 23:34:41.579172 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:34:41.579143 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d08b131_79b5_4235_90f6_44b10f68a36f.slice/crio-b1b5db105a55256b8bd8947fee37f221a2ad7cfb58fef9875aff0419b18c10aa WatchSource:0}: Error finding container b1b5db105a55256b8bd8947fee37f221a2ad7cfb58fef9875aff0419b18c10aa: Status 404 returned error can't find the container with id b1b5db105a55256b8bd8947fee37f221a2ad7cfb58fef9875aff0419b18c10aa Sep 17 23:34:41.699762 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:41.699729 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-a6a3d6" event={"ID":"1d08b131-79b5-4235-90f6-44b10f68a36f","Type":"ContainerStarted","Data":"986a3bb337ae2fcdca5542832c9b18f660138d1458ca0eda06c6d561e37c6699"} Sep 17 23:34:41.699953 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:41.699774 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-a6a3d6" event={"ID":"1d08b131-79b5-4235-90f6-44b10f68a36f","Type":"ContainerStarted","Data":"b1b5db105a55256b8bd8947fee37f221a2ad7cfb58fef9875aff0419b18c10aa"} Sep 17 23:34:41.714911 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:41.714867 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7853-a6a3d6" podStartSLOduration=1.714853158 podStartE2EDuration="1.714853158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:41.714590024 +0000 UTC m=+1709.832069530" watchObservedRunningTime="2026-09-17 23:34:41.714853158 +0000 UTC m=+1709.832332608" Sep 17 23:34:42.328356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:42.328325 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7853-a6a3d6"] Sep 17 23:34:42.337085 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:42.337062 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7853-a6a3d6"] Sep 17 23:34:42.451910 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:42.451862 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7853-fb8638"] Sep 17 23:34:42.452286 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:42.452272 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d08b131-79b5-4235-90f6-44b10f68a36f" containerName="test-curl-7853-a6a3d6" Sep 17 23:34:42.452345 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:42.452288 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d08b131-79b5-4235-90f6-44b10f68a36f" containerName="test-curl-7853-a6a3d6" Sep 17 23:34:42.452387 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:42.452348 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d08b131-79b5-4235-90f6-44b10f68a36f" containerName="test-curl-7853-a6a3d6" Sep 17 23:34:42.455467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:42.455452 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-fb8638" Sep 17 23:34:42.461749 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:42.461726 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-fb8638"] Sep 17 23:34:42.552663 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:42.552634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g8f7\" (UniqueName: \"kubernetes.io/projected/da1e7fe7-2f13-4e49-aa18-45b0ae83252a-kube-api-access-2g8f7\") pod \"test-curl-7853-fb8638\" (UID: \"da1e7fe7-2f13-4e49-aa18-45b0ae83252a\") " pod="opendatahub/test-curl-7853-fb8638" Sep 17 23:34:42.654054 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:42.653966 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g8f7\" (UniqueName: \"kubernetes.io/projected/da1e7fe7-2f13-4e49-aa18-45b0ae83252a-kube-api-access-2g8f7\") pod \"test-curl-7853-fb8638\" (UID: \"da1e7fe7-2f13-4e49-aa18-45b0ae83252a\") " pod="opendatahub/test-curl-7853-fb8638" Sep 17 23:34:42.662688 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:42.662663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g8f7\" (UniqueName: \"kubernetes.io/projected/da1e7fe7-2f13-4e49-aa18-45b0ae83252a-kube-api-access-2g8f7\") pod \"test-curl-7853-fb8638\" (UID: \"da1e7fe7-2f13-4e49-aa18-45b0ae83252a\") " pod="opendatahub/test-curl-7853-fb8638" Sep 17 23:34:42.704695 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:34:42.704670 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7853-a6a3d6" container="test-curl-7853-a6a3d6" Sep 17 23:34:42.765648 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:42.765627 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-fb8638" Sep 17 23:34:43.035228 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:43.035200 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-fb8638"] Sep 17 23:34:43.037313 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:34:43.037288 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda1e7fe7_2f13_4e49_aa18_45b0ae83252a.slice/crio-c962de681dcd332e98a4babd4a8d229ac10c339b05d33fc00d84d028c1056113 WatchSource:0}: Error finding container c962de681dcd332e98a4babd4a8d229ac10c339b05d33fc00d84d028c1056113: Status 404 returned error can't find the container with id c962de681dcd332e98a4babd4a8d229ac10c339b05d33fc00d84d028c1056113 Sep 17 23:34:43.711200 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:43.711163 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-fb8638" event={"ID":"da1e7fe7-2f13-4e49-aa18-45b0ae83252a","Type":"ContainerStarted","Data":"0bc72c80f8efd40afbf145c52f8148b195253f0b81425ccef03f68e8c8f6f767"} Sep 17 23:34:43.711650 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:43.711208 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-fb8638" event={"ID":"da1e7fe7-2f13-4e49-aa18-45b0ae83252a","Type":"ContainerStarted","Data":"c962de681dcd332e98a4babd4a8d229ac10c339b05d33fc00d84d028c1056113"} Sep 17 23:34:43.711650 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:43.711319 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7853-a6a3d6" podUID="1d08b131-79b5-4235-90f6-44b10f68a36f" containerName="test-curl-7853-a6a3d6" containerID="cri-o://986a3bb337ae2fcdca5542832c9b18f660138d1458ca0eda06c6d561e37c6699" gracePeriod=2 Sep 17 23:34:43.728214 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:34:43.728179 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7853-a6a3d6\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="1d08b131-79b5-4235-90f6-44b10f68a36f" pod="opendatahub/test-curl-7853-a6a3d6" Sep 17 23:34:43.728464 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:43.728428 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7853-fb8638" podStartSLOduration=1.7284139490000001 podStartE2EDuration="1.728413949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:43.726361231 +0000 UTC m=+1711.843840674" watchObservedRunningTime="2026-09-17 23:34:43.728413949 +0000 UTC m=+1711.845893399" Sep 17 23:34:44.285993 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:34:44.285958 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7853-a6a3d6\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="1d08b131-79b5-4235-90f6-44b10f68a36f" pod="opendatahub/test-curl-7853-a6a3d6" Sep 17 23:34:44.295755 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:44.295727 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7853-fb8638"] Sep 17 23:34:44.301630 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:44.301606 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7853-fb8638"] Sep 17 23:34:44.716162 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:34:44.716135 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7853-fb8638" container="test-curl-7853-fb8638" Sep 17 23:34:45.721322 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:45.721290 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7853-fb8638" podUID="da1e7fe7-2f13-4e49-aa18-45b0ae83252a" containerName="test-curl-7853-fb8638" containerID="cri-o://0bc72c80f8efd40afbf145c52f8148b195253f0b81425ccef03f68e8c8f6f767" gracePeriod=2 Sep 17 23:34:45.723548 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:34:45.723512 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7853-fb8638\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="da1e7fe7-2f13-4e49-aa18-45b0ae83252a" pod="opendatahub/test-curl-7853-fb8638" Sep 17 23:34:45.806138 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:45.806106 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:34:45.904782 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:45.904756 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-a6a3d6" Sep 17 23:34:45.907286 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:34:45.907256 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7853-fb8638\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="da1e7fe7-2f13-4e49-aa18-45b0ae83252a" pod="opendatahub/test-curl-7853-fb8638" Sep 17 23:34:45.985167 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:45.985092 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk7lx\" (UniqueName: \"kubernetes.io/projected/1d08b131-79b5-4235-90f6-44b10f68a36f-kube-api-access-tk7lx\") pod \"1d08b131-79b5-4235-90f6-44b10f68a36f\" (UID: \"1d08b131-79b5-4235-90f6-44b10f68a36f\") " Sep 17 23:34:45.987250 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:45.987221 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d08b131-79b5-4235-90f6-44b10f68a36f-kube-api-access-tk7lx" (OuterVolumeSpecName: "kube-api-access-tk7lx") pod "1d08b131-79b5-4235-90f6-44b10f68a36f" (UID: "1d08b131-79b5-4235-90f6-44b10f68a36f"). InnerVolumeSpecName "kube-api-access-tk7lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:34:46.086095 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:46.086065 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk7lx\" (UniqueName: \"kubernetes.io/projected/1d08b131-79b5-4235-90f6-44b10f68a36f-kube-api-access-tk7lx\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:34:46.287619 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:46.287475 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d08b131-79b5-4235-90f6-44b10f68a36f" path="/var/lib/kubelet/pods/1d08b131-79b5-4235-90f6-44b10f68a36f/volumes" Sep 17 23:34:46.726917 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:46.726878 2559 generic.go:356] "Generic (PLEG): container finished" podID="1d08b131-79b5-4235-90f6-44b10f68a36f" containerID="986a3bb337ae2fcdca5542832c9b18f660138d1458ca0eda06c6d561e37c6699" exitCode=137 Sep 17 23:34:46.727302 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:46.726929 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-a6a3d6" Sep 17 23:34:46.727302 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:46.726956 2559 scope.go:117] "RemoveContainer" containerID="986a3bb337ae2fcdca5542832c9b18f660138d1458ca0eda06c6d561e37c6699" Sep 17 23:34:46.735838 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:46.735821 2559 scope.go:117] "RemoveContainer" containerID="986a3bb337ae2fcdca5542832c9b18f660138d1458ca0eda06c6d561e37c6699" Sep 17 23:34:46.736108 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:34:46.736092 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986a3bb337ae2fcdca5542832c9b18f660138d1458ca0eda06c6d561e37c6699\": container with ID starting with 986a3bb337ae2fcdca5542832c9b18f660138d1458ca0eda06c6d561e37c6699 not found: ID does not exist" containerID="986a3bb337ae2fcdca5542832c9b18f660138d1458ca0eda06c6d561e37c6699" Sep 17 23:34:46.736171 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:46.736117 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986a3bb337ae2fcdca5542832c9b18f660138d1458ca0eda06c6d561e37c6699"} err="failed to get container status \"986a3bb337ae2fcdca5542832c9b18f660138d1458ca0eda06c6d561e37c6699\": rpc error: code = NotFound desc = could not find container \"986a3bb337ae2fcdca5542832c9b18f660138d1458ca0eda06c6d561e37c6699\": container with ID starting with 986a3bb337ae2fcdca5542832c9b18f660138d1458ca0eda06c6d561e37c6699 not found: ID does not exist" Sep 17 23:34:47.912877 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:47.912858 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-fb8638" Sep 17 23:34:48.002782 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:48.002705 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g8f7\" (UniqueName: \"kubernetes.io/projected/da1e7fe7-2f13-4e49-aa18-45b0ae83252a-kube-api-access-2g8f7\") pod \"da1e7fe7-2f13-4e49-aa18-45b0ae83252a\" (UID: \"da1e7fe7-2f13-4e49-aa18-45b0ae83252a\") " Sep 17 23:34:48.004785 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:48.004758 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1e7fe7-2f13-4e49-aa18-45b0ae83252a-kube-api-access-2g8f7" (OuterVolumeSpecName: "kube-api-access-2g8f7") pod "da1e7fe7-2f13-4e49-aa18-45b0ae83252a" (UID: "da1e7fe7-2f13-4e49-aa18-45b0ae83252a"). InnerVolumeSpecName "kube-api-access-2g8f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:34:48.103815 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:48.103779 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g8f7\" (UniqueName: \"kubernetes.io/projected/da1e7fe7-2f13-4e49-aa18-45b0ae83252a-kube-api-access-2g8f7\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:34:48.286863 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:48.286768 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da1e7fe7-2f13-4e49-aa18-45b0ae83252a" path="/var/lib/kubelet/pods/da1e7fe7-2f13-4e49-aa18-45b0ae83252a/volumes" Sep 17 23:34:48.740444 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:48.740404 2559 generic.go:356] "Generic (PLEG): container finished" podID="da1e7fe7-2f13-4e49-aa18-45b0ae83252a" containerID="0bc72c80f8efd40afbf145c52f8148b195253f0b81425ccef03f68e8c8f6f767" exitCode=137 Sep 17 23:34:48.740632 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:48.740458 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-fb8638" Sep 17 23:34:48.740632 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:48.740497 2559 scope.go:117] "RemoveContainer" containerID="0bc72c80f8efd40afbf145c52f8148b195253f0b81425ccef03f68e8c8f6f767" Sep 17 23:34:48.749843 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:48.749826 2559 scope.go:117] "RemoveContainer" containerID="0bc72c80f8efd40afbf145c52f8148b195253f0b81425ccef03f68e8c8f6f767" Sep 17 23:34:48.750156 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:34:48.750132 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc72c80f8efd40afbf145c52f8148b195253f0b81425ccef03f68e8c8f6f767\": container with ID starting with 0bc72c80f8efd40afbf145c52f8148b195253f0b81425ccef03f68e8c8f6f767 not found: ID does not exist" containerID="0bc72c80f8efd40afbf145c52f8148b195253f0b81425ccef03f68e8c8f6f767" Sep 17 23:34:48.750273 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:48.750160 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc72c80f8efd40afbf145c52f8148b195253f0b81425ccef03f68e8c8f6f767"} err="failed to get container status \"0bc72c80f8efd40afbf145c52f8148b195253f0b81425ccef03f68e8c8f6f767\": rpc error: code = NotFound desc = could not find container \"0bc72c80f8efd40afbf145c52f8148b195253f0b81425ccef03f68e8c8f6f767\": container with ID starting with 0bc72c80f8efd40afbf145c52f8148b195253f0b81425ccef03f68e8c8f6f767 not found: ID does not exist" Sep 17 23:34:55.899138 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:34:55.899100 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:35:04.591203 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:35:04.591169 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:35:34.104712 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:35:34.104670 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:36:00.511551 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:00.511472 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:36:09.798497 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:09.798459 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:36:14.331247 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:14.331218 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:36:14.337367 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:14.337348 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:36:15.211184 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:15.211157 2559 scope.go:117] "RemoveContainer" containerID="b084b2822ee4d7825c1d5d1d35c1b193799ad0a162797c1454d9ccbae3825fd6" Sep 17 23:36:15.219553 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:15.219530 2559 scope.go:117] "RemoveContainer" containerID="f841cd78390c51cf7019c1bf0ec1445b36ec05bbbbd71292d3e2f71d2e06c369" Sep 17 23:36:16.331150 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.331113 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq"] Sep 17 23:36:16.331653 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.331602 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1e7fe7-2f13-4e49-aa18-45b0ae83252a" containerName="test-curl-7853-fb8638" Sep 17 23:36:16.331653 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.331618 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1e7fe7-2f13-4e49-aa18-45b0ae83252a" containerName="test-curl-7853-fb8638" Sep 17 23:36:16.331787 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.331709 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="da1e7fe7-2f13-4e49-aa18-45b0ae83252a" containerName="test-curl-7853-fb8638" Sep 17 23:36:16.336163 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.336141 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.338397 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.338380 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-36fb5c-openshift-default-dockercfg-5tshb\"" Sep 17 23:36:16.347331 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.347307 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq"] Sep 17 23:36:16.480127 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.480086 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c549a62-a914-4595-8507-ca2a9000b709-istio-podinfo\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.480309 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.480134 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-istio-data\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.480309 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.480209 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-istio-envoy\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.480309 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.480239 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c549a62-a914-4595-8507-ca2a9000b709-istio-token\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.480309 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.480267 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c549a62-a914-4595-8507-ca2a9000b709-istiod-ca-cert\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.480309 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.480297 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pskzq\" (UniqueName: \"kubernetes.io/projected/5c549a62-a914-4595-8507-ca2a9000b709-kube-api-access-pskzq\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.480539 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.480329 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-credential-socket\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.480539 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.480351 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-workload-certs\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.480539 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.480422 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-workload-socket\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.581475 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.581377 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-istio-envoy\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.581475 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.581416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c549a62-a914-4595-8507-ca2a9000b709-istio-token\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.581475 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.581447 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c549a62-a914-4595-8507-ca2a9000b709-istiod-ca-cert\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.581758 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.581491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pskzq\" (UniqueName: \"kubernetes.io/projected/5c549a62-a914-4595-8507-ca2a9000b709-kube-api-access-pskzq\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.581758 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.581527 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-credential-socket\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.581758 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.581548 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-workload-certs\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.581758 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.581584 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-workload-socket\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.581758 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.581667 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c549a62-a914-4595-8507-ca2a9000b709-istio-podinfo\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.581758 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.581696 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-istio-data\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.582111 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.581947 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-workload-certs\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.582111 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.582007 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-credential-socket\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.582111 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.582033 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-workload-socket\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.582276 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.582130 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-istio-data\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.582550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.582527 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c549a62-a914-4595-8507-ca2a9000b709-istiod-ca-cert\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.584006 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.583977 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-istio-envoy\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.584215 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.584197 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c549a62-a914-4595-8507-ca2a9000b709-istio-podinfo\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.589464 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.589442 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c549a62-a914-4595-8507-ca2a9000b709-istio-token\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.589601 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.589584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pskzq\" (UniqueName: \"kubernetes.io/projected/5c549a62-a914-4595-8507-ca2a9000b709-kube-api-access-pskzq\") pod \"e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.648353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.648316 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:16.924745 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.924716 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq"] Sep 17 23:36:16.927036 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:36:16.927011 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c549a62_a914_4595_8507_ca2a9000b709.slice/crio-5868999a02118f71a685aefd339524c4dbeb76924d40bc843a543e9b80d84de2 WatchSource:0}: Error finding container 5868999a02118f71a685aefd339524c4dbeb76924d40bc843a543e9b80d84de2: Status 404 returned error can't find the container with id 5868999a02118f71a685aefd339524c4dbeb76924d40bc843a543e9b80d84de2 Sep 17 23:36:16.929699 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.929662 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:36:16.929802 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.929745 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:36:16.929802 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:16.929791 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:36:17.208496 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:17.208457 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" event={"ID":"5c549a62-a914-4595-8507-ca2a9000b709","Type":"ContainerStarted","Data":"9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022"} Sep 17 23:36:17.208682 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:17.208513 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" event={"ID":"5c549a62-a914-4595-8507-ca2a9000b709","Type":"ContainerStarted","Data":"5868999a02118f71a685aefd339524c4dbeb76924d40bc843a543e9b80d84de2"} Sep 17 23:36:17.229442 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:17.229394 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" podStartSLOduration=1.229378831 podStartE2EDuration="1.229378831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:17.228730159 +0000 UTC m=+1805.346209629" watchObservedRunningTime="2026-09-17 23:36:17.229378831 +0000 UTC m=+1805.346858285" Sep 17 23:36:17.649303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:17.649201 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:17.654856 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:17.654825 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:18.213595 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:18.213570 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:18.214488 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:18.214467 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:20.606637 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:20.606599 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:36:21.017216 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.017178 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9"] Sep 17 23:36:21.021180 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.021148 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.023685 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.023660 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-36fb5c-serving-cert\"" Sep 17 23:36:21.032958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.032865 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9"] Sep 17 23:36:21.122830 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.122791 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb90830-690f-402d-85bf-e44575d5132d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.122830 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.122852 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxrbk\" (UniqueName: \"kubernetes.io/projected/6cb90830-690f-402d-85bf-e44575d5132d-kube-api-access-gxrbk\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.123178 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.122983 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.123178 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.123072 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-metrics-certs\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.223919 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.223871 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.224130 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.223981 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-metrics-certs\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.224130 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:36:21.224025 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-serving-cert: secret "maas-api-e2e-ipp-a-36fb5c-serving-cert" not found Sep 17 23:36:21.224130 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.224060 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb90830-690f-402d-85bf-e44575d5132d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.224130 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:36:21.224102 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-maas-api-tls podName:6cb90830-690f-402d-85bf-e44575d5132d nodeName:}" failed. No retries permitted until 2026-09-17 23:36:21.724081392 +0000 UTC m=+1809.841560820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-maas-api-tls") pod "maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" (UID: "6cb90830-690f-402d-85bf-e44575d5132d") : secret "maas-api-e2e-ipp-a-36fb5c-serving-cert" not found Sep 17 23:36:21.224376 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.224152 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxrbk\" (UniqueName: \"kubernetes.io/projected/6cb90830-690f-402d-85bf-e44575d5132d-kube-api-access-gxrbk\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.224793 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.224771 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb90830-690f-402d-85bf-e44575d5132d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.226815 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.226792 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-metrics-certs\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.232888 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.232866 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxrbk\" (UniqueName: \"kubernetes.io/projected/6cb90830-690f-402d-85bf-e44575d5132d-kube-api-access-gxrbk\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.729213 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:21.729177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:21.729688 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:36:21.729328 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-serving-cert: secret "maas-api-e2e-ipp-a-36fb5c-serving-cert" not found Sep 17 23:36:21.729688 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:36:21.729390 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-maas-api-tls podName:6cb90830-690f-402d-85bf-e44575d5132d nodeName:}" failed. No retries permitted until 2026-09-17 23:36:22.729375391 +0000 UTC m=+1810.846854835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-maas-api-tls") pod "maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" (UID: "6cb90830-690f-402d-85bf-e44575d5132d") : secret "maas-api-e2e-ipp-a-36fb5c-serving-cert" not found Sep 17 23:36:22.739324 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:22.739284 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:22.741886 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:22.741859 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:22.834624 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:22.834591 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:23.111060 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:23.111035 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9"] Sep 17 23:36:23.113294 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:36:23.113263 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb90830_690f_402d_85bf_e44575d5132d.slice/crio-9f19efad0334187f8cc68fba92c9779191c449552cd8bf3e87c537317e415acc WatchSource:0}: Error finding container 9f19efad0334187f8cc68fba92c9779191c449552cd8bf3e87c537317e415acc: Status 404 returned error can't find the container with id 9f19efad0334187f8cc68fba92c9779191c449552cd8bf3e87c537317e415acc Sep 17 23:36:23.114714 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:23.114693 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 23:36:23.240085 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:23.240042 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" event={"ID":"6cb90830-690f-402d-85bf-e44575d5132d","Type":"ContainerStarted","Data":"9f19efad0334187f8cc68fba92c9779191c449552cd8bf3e87c537317e415acc"} Sep 17 23:36:24.248403 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:24.248356 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" event={"ID":"6cb90830-690f-402d-85bf-e44575d5132d","Type":"ContainerStarted","Data":"cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6"} Sep 17 23:36:24.248821 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:24.248538 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:24.268071 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:24.268031 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" podStartSLOduration=2.747042188 podStartE2EDuration="3.268017844s" totalImagesPullingTime="520.975656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:24.266173114 +0000 UTC m=+1812.383652575" watchObservedRunningTime="2026-09-17 23:36:24.268017844 +0000 UTC m=+1812.385497293" Sep 17 23:36:30.269238 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:30.269206 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:36:32.105089 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:32.105049 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:36:37.326072 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:37.326039 2559 generic.go:356] "Generic (PLEG): container finished" podID="5c549a62-a914-4595-8507-ca2a9000b709" containerID="9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022" exitCode=0 Sep 17 23:36:37.326559 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:37.326112 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" event={"ID":"5c549a62-a914-4595-8507-ca2a9000b709","Type":"ContainerDied","Data":"9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022"} Sep 17 23:36:37.326559 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:37.326496 2559 scope.go:117] "RemoveContainer" containerID="9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022" Sep 17 23:36:37.327528 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:37.327494 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:36:37.327646 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:37.327624 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:36:37.327712 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:37.327679 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 17 23:36:38.333424 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:38.333390 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" event={"ID":"5c549a62-a914-4595-8507-ca2a9000b709","Type":"ContainerStarted","Data":"bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106"} Sep 17 23:36:38.648929 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:38.648816 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:38.649167 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:38.649136 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Sep 17 23:36:38.649255 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:38.649231 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Sep 17 23:36:39.649014 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:39.648972 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Sep 17 23:36:39.649621 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:39.649585 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Sep 17 23:36:40.649331 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:40.649292 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Sep 17 23:36:40.649790 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:40.649375 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Sep 17 23:36:41.649260 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:41.649215 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Sep 17 23:36:41.649475 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:41.649301 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Sep 17 23:36:42.649551 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:42.649510 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Sep 17 23:36:42.650031 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:42.649593 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Sep 17 23:36:43.649443 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:43.649407 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Sep 17 23:36:43.649639 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:43.649468 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Sep 17 23:36:44.653395 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:44.653362 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:44.653749 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:44.653700 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:44.654297 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:44.654277 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:36:51.555043 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.555007 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685"] Sep 17 23:36:51.558532 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.558508 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" Sep 17 23:36:51.561056 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.561035 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3c1ddb-dockercfg-czq4f\"" Sep 17 23:36:51.561155 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.561035 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3c1ddb\"" Sep 17 23:36:51.569841 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.569818 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685"] Sep 17 23:36:51.581454 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.581432 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7"] Sep 17 23:36:51.584037 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.584019 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" Sep 17 23:36:51.597570 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.597543 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7"] Sep 17 23:36:51.696851 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.696817 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91e1518-e7f4-4c49-8fa7-f17698357a34-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7\" (UID: \"d91e1518-e7f4-4c49-8fa7-f17698357a34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" Sep 17 23:36:51.697030 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.696874 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2m5x\" (UniqueName: \"kubernetes.io/projected/d91e1518-e7f4-4c49-8fa7-f17698357a34-kube-api-access-f2m5x\") pod \"payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7\" (UID: \"d91e1518-e7f4-4c49-8fa7-f17698357a34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" Sep 17 23:36:51.697030 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.696922 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a654e8f2-501c-4b82-8b95-77a063552922-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685\" (UID: \"a654e8f2-501c-4b82-8b95-77a063552922\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" Sep 17 23:36:51.697030 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.696972 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49v49\" (UniqueName: \"kubernetes.io/projected/a654e8f2-501c-4b82-8b95-77a063552922-kube-api-access-49v49\") pod \"payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685\" (UID: \"a654e8f2-501c-4b82-8b95-77a063552922\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" Sep 17 23:36:51.798214 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.798180 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49v49\" (UniqueName: \"kubernetes.io/projected/a654e8f2-501c-4b82-8b95-77a063552922-kube-api-access-49v49\") pod \"payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685\" (UID: \"a654e8f2-501c-4b82-8b95-77a063552922\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" Sep 17 23:36:51.798392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.798239 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91e1518-e7f4-4c49-8fa7-f17698357a34-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7\" (UID: \"d91e1518-e7f4-4c49-8fa7-f17698357a34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" Sep 17 23:36:51.798392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.798275 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2m5x\" (UniqueName: \"kubernetes.io/projected/d91e1518-e7f4-4c49-8fa7-f17698357a34-kube-api-access-f2m5x\") pod \"payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7\" (UID: \"d91e1518-e7f4-4c49-8fa7-f17698357a34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" Sep 17 23:36:51.798392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.798293 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a654e8f2-501c-4b82-8b95-77a063552922-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685\" (UID: \"a654e8f2-501c-4b82-8b95-77a063552922\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" Sep 17 23:36:51.799009 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.798990 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a654e8f2-501c-4b82-8b95-77a063552922-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685\" (UID: \"a654e8f2-501c-4b82-8b95-77a063552922\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" Sep 17 23:36:51.799009 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.799001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91e1518-e7f4-4c49-8fa7-f17698357a34-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7\" (UID: \"d91e1518-e7f4-4c49-8fa7-f17698357a34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" Sep 17 23:36:51.807096 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.807044 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2m5x\" (UniqueName: \"kubernetes.io/projected/d91e1518-e7f4-4c49-8fa7-f17698357a34-kube-api-access-f2m5x\") pod \"payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7\" (UID: \"d91e1518-e7f4-4c49-8fa7-f17698357a34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" Sep 17 23:36:51.807167 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.807091 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49v49\" (UniqueName: \"kubernetes.io/projected/a654e8f2-501c-4b82-8b95-77a063552922-kube-api-access-49v49\") pod \"payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685\" (UID: \"a654e8f2-501c-4b82-8b95-77a063552922\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" Sep 17 23:36:51.869231 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.869199 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" Sep 17 23:36:51.896611 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:51.896586 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" Sep 17 23:36:52.142549 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:52.142524 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685"] Sep 17 23:36:52.145477 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:36:52.145445 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda654e8f2_501c_4b82_8b95_77a063552922.slice/crio-c337b5fee17fba5f8b1a713ca15cc39bd7fa254eec81d0a8f2e4e47795c8e2da WatchSource:0}: Error finding container c337b5fee17fba5f8b1a713ca15cc39bd7fa254eec81d0a8f2e4e47795c8e2da: Status 404 returned error can't find the container with id c337b5fee17fba5f8b1a713ca15cc39bd7fa254eec81d0a8f2e4e47795c8e2da Sep 17 23:36:52.181318 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:52.181291 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7"] Sep 17 23:36:52.182932 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:36:52.182891 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91e1518_e7f4_4c49_8fa7_f17698357a34.slice/crio-26622573ffe031fc831eed564f8b18a722907a26b85cdebc04b49822febaa051 WatchSource:0}: Error finding container 26622573ffe031fc831eed564f8b18a722907a26b85cdebc04b49822febaa051: Status 404 returned error can't find the container with id 26622573ffe031fc831eed564f8b18a722907a26b85cdebc04b49822febaa051 Sep 17 23:36:52.429326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:52.429229 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" event={"ID":"d91e1518-e7f4-4c49-8fa7-f17698357a34","Type":"ContainerStarted","Data":"0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9"} Sep 17 23:36:52.429326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:52.429286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" event={"ID":"d91e1518-e7f4-4c49-8fa7-f17698357a34","Type":"ContainerStarted","Data":"26622573ffe031fc831eed564f8b18a722907a26b85cdebc04b49822febaa051"} Sep 17 23:36:52.429326 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:52.429307 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" Sep 17 23:36:52.430860 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:52.430835 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" event={"ID":"a654e8f2-501c-4b82-8b95-77a063552922","Type":"ContainerStarted","Data":"d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12"} Sep 17 23:36:52.430986 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:52.430883 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" event={"ID":"a654e8f2-501c-4b82-8b95-77a063552922","Type":"ContainerStarted","Data":"c337b5fee17fba5f8b1a713ca15cc39bd7fa254eec81d0a8f2e4e47795c8e2da"} Sep 17 23:36:52.430986 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:52.430952 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" Sep 17 23:36:52.451194 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:52.451140 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" podStartSLOduration=1.451118944 podStartE2EDuration="1.451118944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:52.448268648 +0000 UTC m=+1840.565748155" watchObservedRunningTime="2026-09-17 23:36:52.451118944 +0000 UTC m=+1840.568598396" Sep 17 23:36:54.899999 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:54.899947 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" podStartSLOduration=3.899928234 podStartE2EDuration="3.899928234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:52.467180056 +0000 UTC m=+1840.584659518" watchObservedRunningTime="2026-09-17 23:36:54.899928234 +0000 UTC m=+1843.017407683" Sep 17 23:36:54.902928 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:36:54.902562 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:37:03.439095 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:37:03.439062 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" Sep 17 23:37:03.439515 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:37:03.439367 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" Sep 17 23:37:18.555307 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:37:18.555276 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x5r9q_c2f4d315-2130-4dc1-8c78-fca47740b436/payload-processing/0.log" Sep 17 23:37:18.677763 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:37:18.677732 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7_d91e1518-e7f4-4c49-8fa7-f17698357a34/payload-processing/0.log" Sep 17 23:37:27.500047 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:37:27.499954 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:37:48.403356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:37:48.403322 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:37:54.818566 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:37:54.818514 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x5r9q_c2f4d315-2130-4dc1-8c78-fca47740b436/payload-processing/0.log" Sep 17 23:37:54.938610 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:37:54.938582 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7_d91e1518-e7f4-4c49-8fa7-f17698357a34/payload-processing/0.log" Sep 17 23:37:55.067307 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:37:55.067276 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7_d91e1518-e7f4-4c49-8fa7-f17698357a34/payload-processing/0.log" Sep 17 23:38:02.493045 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:02.493009 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn"] Sep 17 23:38:02.497067 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:02.497051 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" Sep 17 23:38:02.500745 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:02.500722 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-26b61a-dockercfg-9bmph\"" Sep 17 23:38:02.501702 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:02.501684 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-26b61a\"" Sep 17 23:38:02.512465 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:02.512437 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn"] Sep 17 23:38:02.637687 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:02.637647 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e147edd-8d8f-4258-b579-74a47d20d8d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn\" (UID: \"0e147edd-8d8f-4258-b579-74a47d20d8d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" Sep 17 23:38:02.637861 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:02.637740 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h8c7\" (UniqueName: \"kubernetes.io/projected/0e147edd-8d8f-4258-b579-74a47d20d8d2-kube-api-access-8h8c7\") pod \"payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn\" (UID: \"0e147edd-8d8f-4258-b579-74a47d20d8d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" Sep 17 23:38:02.739076 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:02.739038 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8c7\" (UniqueName: \"kubernetes.io/projected/0e147edd-8d8f-4258-b579-74a47d20d8d2-kube-api-access-8h8c7\") pod \"payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn\" (UID: \"0e147edd-8d8f-4258-b579-74a47d20d8d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" Sep 17 23:38:02.739244 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:02.739123 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e147edd-8d8f-4258-b579-74a47d20d8d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn\" (UID: \"0e147edd-8d8f-4258-b579-74a47d20d8d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" Sep 17 23:38:02.739820 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:02.739799 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e147edd-8d8f-4258-b579-74a47d20d8d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn\" (UID: \"0e147edd-8d8f-4258-b579-74a47d20d8d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" Sep 17 23:38:02.747032 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:02.746970 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8c7\" (UniqueName: \"kubernetes.io/projected/0e147edd-8d8f-4258-b579-74a47d20d8d2-kube-api-access-8h8c7\") pod \"payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn\" (UID: \"0e147edd-8d8f-4258-b579-74a47d20d8d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" Sep 17 23:38:02.808034 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:02.808003 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" Sep 17 23:38:03.084253 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:03.084226 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn"] Sep 17 23:38:03.086631 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:38:03.086606 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e147edd_8d8f_4258_b579_74a47d20d8d2.slice/crio-1b36e67d6ae6e8c56205dd3d47883da294ab5e3954fbe6c2aafbf2be60969b90 WatchSource:0}: Error finding container 1b36e67d6ae6e8c56205dd3d47883da294ab5e3954fbe6c2aafbf2be60969b90: Status 404 returned error can't find the container with id 1b36e67d6ae6e8c56205dd3d47883da294ab5e3954fbe6c2aafbf2be60969b90 Sep 17 23:38:03.816514 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:03.816476 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" event={"ID":"0e147edd-8d8f-4258-b579-74a47d20d8d2","Type":"ContainerStarted","Data":"69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d"} Sep 17 23:38:03.816937 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:03.816536 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" event={"ID":"0e147edd-8d8f-4258-b579-74a47d20d8d2","Type":"ContainerStarted","Data":"1b36e67d6ae6e8c56205dd3d47883da294ab5e3954fbe6c2aafbf2be60969b90"} Sep 17 23:38:03.816937 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:03.816690 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" Sep 17 23:38:03.856176 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:03.856118 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" podStartSLOduration=1.856098308 podStartE2EDuration="1.856098308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:38:03.855986415 +0000 UTC m=+1911.973465894" watchObservedRunningTime="2026-09-17 23:38:03.856098308 +0000 UTC m=+1911.973577763" Sep 17 23:38:07.400639 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:07.400599 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:38:14.824267 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:14.824233 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" Sep 17 23:38:40.724734 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:40.724689 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn"] Sep 17 23:38:40.725335 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:40.724934 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" podUID="0e147edd-8d8f-4258-b579-74a47d20d8d2" containerName="payload-pre-processing" containerID="cri-o://69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d" gracePeriod=30 Sep 17 23:38:40.978295 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:40.978234 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" Sep 17 23:38:41.010202 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.010169 2559 generic.go:356] "Generic (PLEG): container finished" podID="0e147edd-8d8f-4258-b579-74a47d20d8d2" containerID="69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d" exitCode=0 Sep 17 23:38:41.010202 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.010208 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" Sep 17 23:38:41.010412 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.010235 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" event={"ID":"0e147edd-8d8f-4258-b579-74a47d20d8d2","Type":"ContainerDied","Data":"69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d"} Sep 17 23:38:41.010412 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.010290 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn" event={"ID":"0e147edd-8d8f-4258-b579-74a47d20d8d2","Type":"ContainerDied","Data":"1b36e67d6ae6e8c56205dd3d47883da294ab5e3954fbe6c2aafbf2be60969b90"} Sep 17 23:38:41.010412 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.010319 2559 scope.go:117] "RemoveContainer" containerID="69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d" Sep 17 23:38:41.019277 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.019256 2559 scope.go:117] "RemoveContainer" containerID="69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d" Sep 17 23:38:41.019606 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:38:41.019577 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d\": container with ID starting with 69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d not found: ID does not exist" containerID="69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d" Sep 17 23:38:41.019702 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.019607 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d"} err="failed to get container status \"69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d\": rpc error: code = NotFound desc = could not find container \"69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d\": container with ID starting with 69bdd74d1943d8e32dfe49dc30c91167dfb423b29e5a60cfdfb9075ebebb420d not found: ID does not exist" Sep 17 23:38:41.076974 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.076946 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h8c7\" (UniqueName: \"kubernetes.io/projected/0e147edd-8d8f-4258-b579-74a47d20d8d2-kube-api-access-8h8c7\") pod \"0e147edd-8d8f-4258-b579-74a47d20d8d2\" (UID: \"0e147edd-8d8f-4258-b579-74a47d20d8d2\") " Sep 17 23:38:41.077131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.077025 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e147edd-8d8f-4258-b579-74a47d20d8d2-plugins-config-volume\") pod \"0e147edd-8d8f-4258-b579-74a47d20d8d2\" (UID: \"0e147edd-8d8f-4258-b579-74a47d20d8d2\") " Sep 17 23:38:41.077409 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.077381 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e147edd-8d8f-4258-b579-74a47d20d8d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e147edd-8d8f-4258-b579-74a47d20d8d2" (UID: "0e147edd-8d8f-4258-b579-74a47d20d8d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:38:41.079217 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.079194 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e147edd-8d8f-4258-b579-74a47d20d8d2-kube-api-access-8h8c7" (OuterVolumeSpecName: "kube-api-access-8h8c7") pod "0e147edd-8d8f-4258-b579-74a47d20d8d2" (UID: "0e147edd-8d8f-4258-b579-74a47d20d8d2"). InnerVolumeSpecName "kube-api-access-8h8c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:38:41.177979 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.177955 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h8c7\" (UniqueName: \"kubernetes.io/projected/0e147edd-8d8f-4258-b579-74a47d20d8d2-kube-api-access-8h8c7\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:38:41.177979 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.177977 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e147edd-8d8f-4258-b579-74a47d20d8d2-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:38:41.331789 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.331758 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn"] Sep 17 23:38:41.337933 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:41.337891 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26b61a-6f7d7494f8-sspvn"] Sep 17 23:38:42.288146 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:42.288114 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e147edd-8d8f-4258-b579-74a47d20d8d2" path="/var/lib/kubelet/pods/0e147edd-8d8f-4258-b579-74a47d20d8d2/volumes" Sep 17 23:38:44.900111 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:38:44.900081 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:39:04.001527 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:04.001445 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:39:06.743392 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:06.743360 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9"] Sep 17 23:39:06.743877 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:06.743576 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" podUID="6cb90830-690f-402d-85bf-e44575d5132d" containerName="maas-api" containerID="cri-o://cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6" gracePeriod=30 Sep 17 23:39:07.008269 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.008215 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:39:07.110172 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.110146 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb90830-690f-402d-85bf-e44575d5132d-openshift-service-ca\") pod \"6cb90830-690f-402d-85bf-e44575d5132d\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " Sep 17 23:39:07.110317 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.110190 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxrbk\" (UniqueName: \"kubernetes.io/projected/6cb90830-690f-402d-85bf-e44575d5132d-kube-api-access-gxrbk\") pod \"6cb90830-690f-402d-85bf-e44575d5132d\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " Sep 17 23:39:07.110317 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.110213 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-maas-api-tls\") pod \"6cb90830-690f-402d-85bf-e44575d5132d\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " Sep 17 23:39:07.110317 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.110287 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-metrics-certs\") pod \"6cb90830-690f-402d-85bf-e44575d5132d\" (UID: \"6cb90830-690f-402d-85bf-e44575d5132d\") " Sep 17 23:39:07.110649 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.110618 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb90830-690f-402d-85bf-e44575d5132d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6cb90830-690f-402d-85bf-e44575d5132d" (UID: "6cb90830-690f-402d-85bf-e44575d5132d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:39:07.112306 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.112274 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6cb90830-690f-402d-85bf-e44575d5132d" (UID: "6cb90830-690f-402d-85bf-e44575d5132d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:39:07.112406 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.112314 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6cb90830-690f-402d-85bf-e44575d5132d" (UID: "6cb90830-690f-402d-85bf-e44575d5132d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:39:07.112406 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.112336 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb90830-690f-402d-85bf-e44575d5132d-kube-api-access-gxrbk" (OuterVolumeSpecName: "kube-api-access-gxrbk") pod "6cb90830-690f-402d-85bf-e44575d5132d" (UID: "6cb90830-690f-402d-85bf-e44575d5132d"). InnerVolumeSpecName "kube-api-access-gxrbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:39:07.149653 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.149619 2559 generic.go:356] "Generic (PLEG): container finished" podID="6cb90830-690f-402d-85bf-e44575d5132d" containerID="cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6" exitCode=0 Sep 17 23:39:07.149818 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.149675 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" event={"ID":"6cb90830-690f-402d-85bf-e44575d5132d","Type":"ContainerDied","Data":"cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6"} Sep 17 23:39:07.149818 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.149726 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" event={"ID":"6cb90830-690f-402d-85bf-e44575d5132d","Type":"ContainerDied","Data":"9f19efad0334187f8cc68fba92c9779191c449552cd8bf3e87c537317e415acc"} Sep 17 23:39:07.149818 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.149751 2559 scope.go:117] "RemoveContainer" containerID="cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6" Sep 17 23:39:07.150021 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.149666 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9" Sep 17 23:39:07.163798 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.163777 2559 scope.go:117] "RemoveContainer" containerID="cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6" Sep 17 23:39:07.164160 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:39:07.164141 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6\": container with ID starting with cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6 not found: ID does not exist" containerID="cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6" Sep 17 23:39:07.164229 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.164169 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6"} err="failed to get container status \"cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6\": rpc error: code = NotFound desc = could not find container \"cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6\": container with ID starting with cdd5fbfd6f798e821be90a1027398050db037f2864038ef75eb839d0a0f7d7e6 not found: ID does not exist" Sep 17 23:39:07.177316 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.177291 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9"] Sep 17 23:39:07.181809 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.181783 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36fb5c-7b5c8b5f4-wz5k9"] Sep 17 23:39:07.211305 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.211283 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb90830-690f-402d-85bf-e44575d5132d-openshift-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:39:07.211305 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.211303 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxrbk\" (UniqueName: \"kubernetes.io/projected/6cb90830-690f-402d-85bf-e44575d5132d-kube-api-access-gxrbk\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:39:07.211436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.211315 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-maas-api-tls\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:39:07.211436 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:07.211324 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb90830-690f-402d-85bf-e44575d5132d-metrics-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:39:08.288147 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:08.288110 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cb90830-690f-402d-85bf-e44575d5132d" path="/var/lib/kubelet/pods/6cb90830-690f-402d-85bf-e44575d5132d/volumes" Sep 17 23:39:15.258810 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:15.258773 2559 scope.go:117] "RemoveContainer" containerID="c7562320ae8c67b950197b7b23eb0931c846e2cc3ab1a455ed9046a3184f7daf" Sep 17 23:39:17.192275 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:39:17.192241 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:40:00.819322 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:00.819282 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq"] Sep 17 23:40:00.819857 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:00.819586 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" containerID="cri-o://bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106" gracePeriod=30 Sep 17 23:40:06.067433 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.067407 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:40:06.117814 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.117738 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-workload-certs\") pod \"5c549a62-a914-4595-8507-ca2a9000b709\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " Sep 17 23:40:06.117814 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.117790 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c549a62-a914-4595-8507-ca2a9000b709-istio-podinfo\") pod \"5c549a62-a914-4595-8507-ca2a9000b709\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " Sep 17 23:40:06.118049 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.117819 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c549a62-a914-4595-8507-ca2a9000b709-istiod-ca-cert\") pod \"5c549a62-a914-4595-8507-ca2a9000b709\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " Sep 17 23:40:06.118049 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.117845 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-workload-socket\") pod \"5c549a62-a914-4595-8507-ca2a9000b709\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " Sep 17 23:40:06.118049 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.117939 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c549a62-a914-4595-8507-ca2a9000b709-istio-token\") pod \"5c549a62-a914-4595-8507-ca2a9000b709\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " Sep 17 23:40:06.118049 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.117973 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-istio-envoy\") pod \"5c549a62-a914-4595-8507-ca2a9000b709\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " Sep 17 23:40:06.118049 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.118021 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-istio-data\") pod \"5c549a62-a914-4595-8507-ca2a9000b709\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " Sep 17 23:40:06.118294 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.118048 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5c549a62-a914-4595-8507-ca2a9000b709" (UID: "5c549a62-a914-4595-8507-ca2a9000b709"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:06.118294 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.118064 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pskzq\" (UniqueName: \"kubernetes.io/projected/5c549a62-a914-4595-8507-ca2a9000b709-kube-api-access-pskzq\") pod \"5c549a62-a914-4595-8507-ca2a9000b709\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " Sep 17 23:40:06.118294 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.118103 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-credential-socket\") pod \"5c549a62-a914-4595-8507-ca2a9000b709\" (UID: \"5c549a62-a914-4595-8507-ca2a9000b709\") " Sep 17 23:40:06.118294 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.118236 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5c549a62-a914-4595-8507-ca2a9000b709" (UID: "5c549a62-a914-4595-8507-ca2a9000b709"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:06.118513 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.118310 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-istio-data" (OuterVolumeSpecName: "istio-data") pod "5c549a62-a914-4595-8507-ca2a9000b709" (UID: "5c549a62-a914-4595-8507-ca2a9000b709"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:06.118513 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.118322 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c549a62-a914-4595-8507-ca2a9000b709-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5c549a62-a914-4595-8507-ca2a9000b709" (UID: "5c549a62-a914-4595-8507-ca2a9000b709"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:40:06.118513 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.118440 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:06.118513 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.118458 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:06.118513 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.118471 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c549a62-a914-4595-8507-ca2a9000b709-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:06.118513 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.118483 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:06.118513 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.118485 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5c549a62-a914-4595-8507-ca2a9000b709" (UID: "5c549a62-a914-4595-8507-ca2a9000b709"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:06.120045 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.120019 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c549a62-a914-4595-8507-ca2a9000b709-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5c549a62-a914-4595-8507-ca2a9000b709" (UID: "5c549a62-a914-4595-8507-ca2a9000b709"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:40:06.120160 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.120096 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c549a62-a914-4595-8507-ca2a9000b709-istio-token" (OuterVolumeSpecName: "istio-token") pod "5c549a62-a914-4595-8507-ca2a9000b709" (UID: "5c549a62-a914-4595-8507-ca2a9000b709"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:40:06.120226 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.120206 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c549a62-a914-4595-8507-ca2a9000b709-kube-api-access-pskzq" (OuterVolumeSpecName: "kube-api-access-pskzq") pod "5c549a62-a914-4595-8507-ca2a9000b709" (UID: "5c549a62-a914-4595-8507-ca2a9000b709"). InnerVolumeSpecName "kube-api-access-pskzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:40:06.120266 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.120224 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5c549a62-a914-4595-8507-ca2a9000b709" (UID: "5c549a62-a914-4595-8507-ca2a9000b709"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:06.219099 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.219071 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c549a62-a914-4595-8507-ca2a9000b709-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:06.219099 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.219097 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c549a62-a914-4595-8507-ca2a9000b709-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:06.219268 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.219109 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:06.219268 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.219117 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pskzq\" (UniqueName: \"kubernetes.io/projected/5c549a62-a914-4595-8507-ca2a9000b709-kube-api-access-pskzq\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:06.219268 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.219126 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c549a62-a914-4595-8507-ca2a9000b709-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:06.458978 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.458946 2559 generic.go:356] "Generic (PLEG): container finished" podID="5c549a62-a914-4595-8507-ca2a9000b709" containerID="bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106" exitCode=0 Sep 17 23:40:06.459152 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.459012 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" Sep 17 23:40:06.459152 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.459026 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" event={"ID":"5c549a62-a914-4595-8507-ca2a9000b709","Type":"ContainerDied","Data":"bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106"} Sep 17 23:40:06.459152 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.459096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq" event={"ID":"5c549a62-a914-4595-8507-ca2a9000b709","Type":"ContainerDied","Data":"5868999a02118f71a685aefd339524c4dbeb76924d40bc843a543e9b80d84de2"} Sep 17 23:40:06.459152 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.459124 2559 scope.go:117] "RemoveContainer" containerID="bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106" Sep 17 23:40:06.468463 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.468445 2559 scope.go:117] "RemoveContainer" containerID="9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022" Sep 17 23:40:06.475385 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.475359 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq"] Sep 17 23:40:06.476342 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.476328 2559 scope.go:117] "RemoveContainer" containerID="bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106" Sep 17 23:40:06.476616 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:40:06.476594 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106\": container with ID starting with bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106 not found: ID does not exist" containerID="bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106" Sep 17 23:40:06.476678 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.476629 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106"} err="failed to get container status \"bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106\": rpc error: code = NotFound desc = could not find container \"bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106\": container with ID starting with bbaa14f0b91465902ff852e426f481e530c8cc67dfa6037888e2f6bb09008106 not found: ID does not exist" Sep 17 23:40:06.476678 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.476647 2559 scope.go:117] "RemoveContainer" containerID="9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022" Sep 17 23:40:06.476954 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:40:06.476929 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022\": container with ID starting with 9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022 not found: ID does not exist" containerID="9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022" Sep 17 23:40:06.477067 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.476962 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022"} err="failed to get container status \"9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022\": rpc error: code = NotFound desc = could not find container \"9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022\": container with ID starting with 9f91477f032fbfa22831f0f7f06d7b2726ac95d6489c024ea08aa56425a8b022 not found: ID does not exist" Sep 17 23:40:06.479649 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:06.479631 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-36fb5c-openshift-default-5cf8c45564-mgnvq"] Sep 17 23:40:08.287398 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:08.287368 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c549a62-a914-4595-8507-ca2a9000b709" path="/var/lib/kubelet/pods/5c549a62-a914-4595-8507-ca2a9000b709/volumes" Sep 17 23:40:11.961384 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:11.961350 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7"] Sep 17 23:40:11.961872 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:11.961573 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" podUID="d91e1518-e7f4-4c49-8fa7-f17698357a34" containerName="payload-processing" containerID="cri-o://0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9" gracePeriod=30 Sep 17 23:40:11.974772 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:11.974740 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685"] Sep 17 23:40:11.975216 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:11.975178 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" podUID="a654e8f2-501c-4b82-8b95-77a063552922" containerName="payload-pre-processing" containerID="cri-o://d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12" gracePeriod=30 Sep 17 23:40:12.226590 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.226528 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" Sep 17 23:40:12.257039 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.257013 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" Sep 17 23:40:12.268383 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.268356 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2m5x\" (UniqueName: \"kubernetes.io/projected/d91e1518-e7f4-4c49-8fa7-f17698357a34-kube-api-access-f2m5x\") pod \"d91e1518-e7f4-4c49-8fa7-f17698357a34\" (UID: \"d91e1518-e7f4-4c49-8fa7-f17698357a34\") " Sep 17 23:40:12.268518 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.268504 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91e1518-e7f4-4c49-8fa7-f17698357a34-plugins-config-volume\") pod \"d91e1518-e7f4-4c49-8fa7-f17698357a34\" (UID: \"d91e1518-e7f4-4c49-8fa7-f17698357a34\") " Sep 17 23:40:12.268894 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.268871 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91e1518-e7f4-4c49-8fa7-f17698357a34-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d91e1518-e7f4-4c49-8fa7-f17698357a34" (UID: "d91e1518-e7f4-4c49-8fa7-f17698357a34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:40:12.270611 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.270579 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91e1518-e7f4-4c49-8fa7-f17698357a34-kube-api-access-f2m5x" (OuterVolumeSpecName: "kube-api-access-f2m5x") pod "d91e1518-e7f4-4c49-8fa7-f17698357a34" (UID: "d91e1518-e7f4-4c49-8fa7-f17698357a34"). InnerVolumeSpecName "kube-api-access-f2m5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:40:12.369697 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.369666 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49v49\" (UniqueName: \"kubernetes.io/projected/a654e8f2-501c-4b82-8b95-77a063552922-kube-api-access-49v49\") pod \"a654e8f2-501c-4b82-8b95-77a063552922\" (UID: \"a654e8f2-501c-4b82-8b95-77a063552922\") " Sep 17 23:40:12.369824 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.369781 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a654e8f2-501c-4b82-8b95-77a063552922-plugins-config-volume\") pod \"a654e8f2-501c-4b82-8b95-77a063552922\" (UID: \"a654e8f2-501c-4b82-8b95-77a063552922\") " Sep 17 23:40:12.370121 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.370094 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91e1518-e7f4-4c49-8fa7-f17698357a34-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:12.370121 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.370120 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2m5x\" (UniqueName: \"kubernetes.io/projected/d91e1518-e7f4-4c49-8fa7-f17698357a34-kube-api-access-f2m5x\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:12.370285 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.370180 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a654e8f2-501c-4b82-8b95-77a063552922-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a654e8f2-501c-4b82-8b95-77a063552922" (UID: "a654e8f2-501c-4b82-8b95-77a063552922"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:40:12.371837 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.371815 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a654e8f2-501c-4b82-8b95-77a063552922-kube-api-access-49v49" (OuterVolumeSpecName: "kube-api-access-49v49") pod "a654e8f2-501c-4b82-8b95-77a063552922" (UID: "a654e8f2-501c-4b82-8b95-77a063552922"). InnerVolumeSpecName "kube-api-access-49v49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:40:12.470968 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.470935 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a654e8f2-501c-4b82-8b95-77a063552922-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:12.470968 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.470962 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49v49\" (UniqueName: \"kubernetes.io/projected/a654e8f2-501c-4b82-8b95-77a063552922-kube-api-access-49v49\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:12.490277 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.490213 2559 generic.go:356] "Generic (PLEG): container finished" podID="d91e1518-e7f4-4c49-8fa7-f17698357a34" containerID="0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9" exitCode=0 Sep 17 23:40:12.490277 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.490258 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" Sep 17 23:40:12.490473 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.490305 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" event={"ID":"d91e1518-e7f4-4c49-8fa7-f17698357a34","Type":"ContainerDied","Data":"0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9"} Sep 17 23:40:12.490473 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.490365 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7" event={"ID":"d91e1518-e7f4-4c49-8fa7-f17698357a34","Type":"ContainerDied","Data":"26622573ffe031fc831eed564f8b18a722907a26b85cdebc04b49822febaa051"} Sep 17 23:40:12.490473 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.490393 2559 scope.go:117] "RemoveContainer" containerID="0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9" Sep 17 23:40:12.491648 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.491588 2559 generic.go:356] "Generic (PLEG): container finished" podID="a654e8f2-501c-4b82-8b95-77a063552922" containerID="d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12" exitCode=0 Sep 17 23:40:12.491648 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.491618 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" event={"ID":"a654e8f2-501c-4b82-8b95-77a063552922","Type":"ContainerDied","Data":"d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12"} Sep 17 23:40:12.491648 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.491632 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" Sep 17 23:40:12.491842 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.491665 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685" event={"ID":"a654e8f2-501c-4b82-8b95-77a063552922","Type":"ContainerDied","Data":"c337b5fee17fba5f8b1a713ca15cc39bd7fa254eec81d0a8f2e4e47795c8e2da"} Sep 17 23:40:12.501616 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.501600 2559 scope.go:117] "RemoveContainer" containerID="0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9" Sep 17 23:40:12.501885 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:40:12.501870 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9\": container with ID starting with 0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9 not found: ID does not exist" containerID="0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9" Sep 17 23:40:12.501958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.501891 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9"} err="failed to get container status \"0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9\": rpc error: code = NotFound desc = could not find container \"0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9\": container with ID starting with 0ca5e39fafbf7dddf7070116e779c16814e82ad760224b29e1acb3a1856ca6a9 not found: ID does not exist" Sep 17 23:40:12.501958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.501923 2559 scope.go:117] "RemoveContainer" containerID="d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12" Sep 17 23:40:12.509673 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.509659 2559 scope.go:117] "RemoveContainer" containerID="d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12" Sep 17 23:40:12.509946 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:40:12.509928 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12\": container with ID starting with d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12 not found: ID does not exist" containerID="d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12" Sep 17 23:40:12.510016 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.509951 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12"} err="failed to get container status \"d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12\": rpc error: code = NotFound desc = could not find container \"d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12\": container with ID starting with d4efcac3d8f09456fdcdc13a70a184528b0153dd7289ae1410cc9de3a510db12 not found: ID does not exist" Sep 17 23:40:12.517125 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.517104 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7"] Sep 17 23:40:12.523006 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.522987 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3c1ddb-c4cd49bb5-944f7"] Sep 17 23:40:12.543798 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.543777 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685"] Sep 17 23:40:12.547281 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:12.547260 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3c1ddb-7bc95d6cb6-4f685"] Sep 17 23:40:14.287323 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:14.287282 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a654e8f2-501c-4b82-8b95-77a063552922" path="/var/lib/kubelet/pods/a654e8f2-501c-4b82-8b95-77a063552922/volumes" Sep 17 23:40:14.287819 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:14.287798 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d91e1518-e7f4-4c49-8fa7-f17698357a34" path="/var/lib/kubelet/pods/d91e1518-e7f4-4c49-8fa7-f17698357a34/volumes" Sep 17 23:40:15.304042 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:15.304018 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:40:32.395191 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:32.395114 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:40:36.377061 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:36.377023 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t"] Sep 17 23:40:36.377538 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:36.377274 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerName="istio-proxy" containerID="cri-o://a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735" gracePeriod=30 Sep 17 23:40:41.630128 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.630103 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:40:41.644670 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.644601 2559 generic.go:356] "Generic (PLEG): container finished" podID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerID="a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735" exitCode=0 Sep 17 23:40:41.644670 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.644652 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" Sep 17 23:40:41.644813 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.644680 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" event={"ID":"d7c41495-4ea9-4c22-a74d-b273f369cc31","Type":"ContainerDied","Data":"a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735"} Sep 17 23:40:41.644813 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.644734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t" event={"ID":"d7c41495-4ea9-4c22-a74d-b273f369cc31","Type":"ContainerDied","Data":"9798383f9d6278a0902be320606c75ad0f809542b2151cd76e88979fa121cb64"} Sep 17 23:40:41.644813 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.644766 2559 scope.go:117] "RemoveContainer" containerID="a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735" Sep 17 23:40:41.654465 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.654442 2559 scope.go:117] "RemoveContainer" containerID="dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335" Sep 17 23:40:41.662675 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.662649 2559 scope.go:117] "RemoveContainer" containerID="a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735" Sep 17 23:40:41.662961 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:40:41.662942 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735\": container with ID starting with a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735 not found: ID does not exist" containerID="a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735" Sep 17 23:40:41.663032 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.662970 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735"} err="failed to get container status \"a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735\": rpc error: code = NotFound desc = could not find container \"a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735\": container with ID starting with a0c8eb1b51fe60cf7f3acb3955470e50d2b280d247b29926e74439b4340e5735 not found: ID does not exist" Sep 17 23:40:41.663032 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.662988 2559 scope.go:117] "RemoveContainer" containerID="dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335" Sep 17 23:40:41.663270 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:40:41.663248 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335\": container with ID starting with dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335 not found: ID does not exist" containerID="dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335" Sep 17 23:40:41.663316 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.663280 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335"} err="failed to get container status \"dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335\": rpc error: code = NotFound desc = could not find container \"dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335\": container with ID starting with dd2ae852ef1df062d08d2f3c9f37b14d7a7c929c7643e138a3ce80c62fd77335 not found: ID does not exist" Sep 17 23:40:41.714718 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.714690 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-data\") pod \"d7c41495-4ea9-4c22-a74d-b273f369cc31\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " Sep 17 23:40:41.714857 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.714728 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-podinfo\") pod \"d7c41495-4ea9-4c22-a74d-b273f369cc31\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " Sep 17 23:40:41.714857 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.714755 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-envoy\") pod \"d7c41495-4ea9-4c22-a74d-b273f369cc31\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " Sep 17 23:40:41.714857 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.714807 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-workload-certs\") pod \"d7c41495-4ea9-4c22-a74d-b273f369cc31\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " Sep 17 23:40:41.714857 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.714837 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g255\" (UniqueName: \"kubernetes.io/projected/d7c41495-4ea9-4c22-a74d-b273f369cc31-kube-api-access-6g255\") pod \"d7c41495-4ea9-4c22-a74d-b273f369cc31\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " Sep 17 23:40:41.715104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.714866 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-token\") pod \"d7c41495-4ea9-4c22-a74d-b273f369cc31\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " Sep 17 23:40:41.715104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.714930 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-workload-socket\") pod \"d7c41495-4ea9-4c22-a74d-b273f369cc31\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " Sep 17 23:40:41.715104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.714963 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-credential-socket\") pod \"d7c41495-4ea9-4c22-a74d-b273f369cc31\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " Sep 17 23:40:41.715104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.715012 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c41495-4ea9-4c22-a74d-b273f369cc31-istiod-ca-cert\") pod \"d7c41495-4ea9-4c22-a74d-b273f369cc31\" (UID: \"d7c41495-4ea9-4c22-a74d-b273f369cc31\") " Sep 17 23:40:41.715104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.715020 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-data" (OuterVolumeSpecName: "istio-data") pod "d7c41495-4ea9-4c22-a74d-b273f369cc31" (UID: "d7c41495-4ea9-4c22-a74d-b273f369cc31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:41.715359 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.715156 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d7c41495-4ea9-4c22-a74d-b273f369cc31" (UID: "d7c41495-4ea9-4c22-a74d-b273f369cc31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:41.715359 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.715297 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:41.715359 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.715316 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:41.715509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.715363 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d7c41495-4ea9-4c22-a74d-b273f369cc31" (UID: "d7c41495-4ea9-4c22-a74d-b273f369cc31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:41.715509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.715375 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d7c41495-4ea9-4c22-a74d-b273f369cc31" (UID: "d7c41495-4ea9-4c22-a74d-b273f369cc31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:41.715509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.715466 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c41495-4ea9-4c22-a74d-b273f369cc31-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d7c41495-4ea9-4c22-a74d-b273f369cc31" (UID: "d7c41495-4ea9-4c22-a74d-b273f369cc31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:40:41.717091 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.717063 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d7c41495-4ea9-4c22-a74d-b273f369cc31" (UID: "d7c41495-4ea9-4c22-a74d-b273f369cc31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:40:41.717178 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.717087 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d7c41495-4ea9-4c22-a74d-b273f369cc31" (UID: "d7c41495-4ea9-4c22-a74d-b273f369cc31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:41.717178 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.717108 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c41495-4ea9-4c22-a74d-b273f369cc31-kube-api-access-6g255" (OuterVolumeSpecName: "kube-api-access-6g255") pod "d7c41495-4ea9-4c22-a74d-b273f369cc31" (UID: "d7c41495-4ea9-4c22-a74d-b273f369cc31"). InnerVolumeSpecName "kube-api-access-6g255". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:40:41.717178 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.717116 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-token" (OuterVolumeSpecName: "istio-token") pod "d7c41495-4ea9-4c22-a74d-b273f369cc31" (UID: "d7c41495-4ea9-4c22-a74d-b273f369cc31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:40:41.816655 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.816627 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:41.816655 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.816651 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:41.816799 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.816660 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g255\" (UniqueName: \"kubernetes.io/projected/d7c41495-4ea9-4c22-a74d-b273f369cc31-kube-api-access-6g255\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:41.816799 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.816668 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c41495-4ea9-4c22-a74d-b273f369cc31-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:41.816799 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.816687 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:41.816799 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.816696 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c41495-4ea9-4c22-a74d-b273f369cc31-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:41.816799 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.816704 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c41495-4ea9-4c22-a74d-b273f369cc31-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:41.966087 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.966063 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t"] Sep 17 23:40:41.969727 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:41.969704 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f80c49-openshift-default-5887749d84-mz94t"] Sep 17 23:40:42.286958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:42.286863 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" path="/var/lib/kubelet/pods/d7c41495-4ea9-4c22-a74d-b273f369cc31/volumes" Sep 17 23:40:47.511716 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.511682 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85"] Sep 17 23:40:47.512204 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.511888 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" podUID="8cd59b83-c48d-4c53-9fdd-93850998ef76" containerName="maas-api" containerID="cri-o://6346e6375b69454a23f29cf95761d3fae230b11628511077e7df33e4c737653d" gracePeriod=30 Sep 17 23:40:47.549695 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.549666 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf"] Sep 17 23:40:47.551730 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.551705 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6ac6e-29828130-lqhhf"] Sep 17 23:40:47.587707 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.587620 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq"] Sep 17 23:40:47.587875 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.587850 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" podUID="2842d6c8-adbb-4231-93f4-002f19cd2dd0" containerName="payload-pre-processing" containerID="cri-o://4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b" gracePeriod=30 Sep 17 23:40:47.675727 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.675700 2559 generic.go:356] "Generic (PLEG): container finished" podID="8cd59b83-c48d-4c53-9fdd-93850998ef76" containerID="6346e6375b69454a23f29cf95761d3fae230b11628511077e7df33e4c737653d" exitCode=0 Sep 17 23:40:47.675884 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.675764 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" event={"ID":"8cd59b83-c48d-4c53-9fdd-93850998ef76","Type":"ContainerDied","Data":"6346e6375b69454a23f29cf95761d3fae230b11628511077e7df33e4c737653d"} Sep 17 23:40:47.868625 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.868594 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:40:47.872043 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.872024 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" Sep 17 23:40:47.964745 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.964711 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9thc\" (UniqueName: \"kubernetes.io/projected/2842d6c8-adbb-4231-93f4-002f19cd2dd0-kube-api-access-v9thc\") pod \"2842d6c8-adbb-4231-93f4-002f19cd2dd0\" (UID: \"2842d6c8-adbb-4231-93f4-002f19cd2dd0\") " Sep 17 23:40:47.964942 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.964760 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2842d6c8-adbb-4231-93f4-002f19cd2dd0-plugins-config-volume\") pod \"2842d6c8-adbb-4231-93f4-002f19cd2dd0\" (UID: \"2842d6c8-adbb-4231-93f4-002f19cd2dd0\") " Sep 17 23:40:47.964942 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.964802 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-metrics-certs\") pod \"8cd59b83-c48d-4c53-9fdd-93850998ef76\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " Sep 17 23:40:47.964942 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.964826 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsgfv\" (UniqueName: \"kubernetes.io/projected/8cd59b83-c48d-4c53-9fdd-93850998ef76-kube-api-access-wsgfv\") pod \"8cd59b83-c48d-4c53-9fdd-93850998ef76\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " Sep 17 23:40:47.964942 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.964890 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd59b83-c48d-4c53-9fdd-93850998ef76-openshift-service-ca\") pod \"8cd59b83-c48d-4c53-9fdd-93850998ef76\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " Sep 17 23:40:47.965156 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.964980 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-maas-api-tls\") pod \"8cd59b83-c48d-4c53-9fdd-93850998ef76\" (UID: \"8cd59b83-c48d-4c53-9fdd-93850998ef76\") " Sep 17 23:40:47.965328 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.965300 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2842d6c8-adbb-4231-93f4-002f19cd2dd0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2842d6c8-adbb-4231-93f4-002f19cd2dd0" (UID: "2842d6c8-adbb-4231-93f4-002f19cd2dd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:40:47.965405 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.965326 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd59b83-c48d-4c53-9fdd-93850998ef76-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8cd59b83-c48d-4c53-9fdd-93850998ef76" (UID: "8cd59b83-c48d-4c53-9fdd-93850998ef76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:40:47.966873 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.966850 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2842d6c8-adbb-4231-93f4-002f19cd2dd0-kube-api-access-v9thc" (OuterVolumeSpecName: "kube-api-access-v9thc") pod "2842d6c8-adbb-4231-93f4-002f19cd2dd0" (UID: "2842d6c8-adbb-4231-93f4-002f19cd2dd0"). InnerVolumeSpecName "kube-api-access-v9thc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:40:47.966970 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.966935 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8cd59b83-c48d-4c53-9fdd-93850998ef76" (UID: "8cd59b83-c48d-4c53-9fdd-93850998ef76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:40:47.967313 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.967292 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8cd59b83-c48d-4c53-9fdd-93850998ef76" (UID: "8cd59b83-c48d-4c53-9fdd-93850998ef76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:40:47.967313 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:47.967306 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd59b83-c48d-4c53-9fdd-93850998ef76-kube-api-access-wsgfv" (OuterVolumeSpecName: "kube-api-access-wsgfv") pod "8cd59b83-c48d-4c53-9fdd-93850998ef76" (UID: "8cd59b83-c48d-4c53-9fdd-93850998ef76"). InnerVolumeSpecName "kube-api-access-wsgfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:40:48.066103 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.066039 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-maas-api-tls\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:48.066103 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.066062 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9thc\" (UniqueName: \"kubernetes.io/projected/2842d6c8-adbb-4231-93f4-002f19cd2dd0-kube-api-access-v9thc\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:48.066103 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.066072 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2842d6c8-adbb-4231-93f4-002f19cd2dd0-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:48.066103 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.066080 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd59b83-c48d-4c53-9fdd-93850998ef76-metrics-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:48.066103 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.066089 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsgfv\" (UniqueName: \"kubernetes.io/projected/8cd59b83-c48d-4c53-9fdd-93850998ef76-kube-api-access-wsgfv\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:48.066103 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.066096 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd59b83-c48d-4c53-9fdd-93850998ef76-openshift-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:40:48.287076 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.287045 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c30d6fc-c1e4-4a98-ae57-35030a10004a" path="/var/lib/kubelet/pods/3c30d6fc-c1e4-4a98-ae57-35030a10004a/volumes" Sep 17 23:40:48.682626 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.682594 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" Sep 17 23:40:48.682626 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.682623 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85" event={"ID":"8cd59b83-c48d-4c53-9fdd-93850998ef76","Type":"ContainerDied","Data":"7eb87f09e11f22607cb28e78ae62e3164106fa29fab93456190d863e8fe82843"} Sep 17 23:40:48.683148 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.682674 2559 scope.go:117] "RemoveContainer" containerID="6346e6375b69454a23f29cf95761d3fae230b11628511077e7df33e4c737653d" Sep 17 23:40:48.684182 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.684152 2559 generic.go:356] "Generic (PLEG): container finished" podID="2842d6c8-adbb-4231-93f4-002f19cd2dd0" containerID="4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b" exitCode=0 Sep 17 23:40:48.684303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.684195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" event={"ID":"2842d6c8-adbb-4231-93f4-002f19cd2dd0","Type":"ContainerDied","Data":"4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b"} Sep 17 23:40:48.684303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.684209 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" Sep 17 23:40:48.684303 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.684228 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq" event={"ID":"2842d6c8-adbb-4231-93f4-002f19cd2dd0","Type":"ContainerDied","Data":"7b475b2d84351c43f91d735407fbee09a0d115af9c1ba19f7e2ec9012d8a5091"} Sep 17 23:40:48.691984 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.691961 2559 scope.go:117] "RemoveContainer" containerID="4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b" Sep 17 23:40:48.699777 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.699763 2559 scope.go:117] "RemoveContainer" containerID="4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b" Sep 17 23:40:48.700091 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:40:48.700075 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b\": container with ID starting with 4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b not found: ID does not exist" containerID="4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b" Sep 17 23:40:48.700150 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.700098 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b"} err="failed to get container status \"4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b\": rpc error: code = NotFound desc = could not find container \"4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b\": container with ID starting with 4af08eae5dbac019ce15581d5e8a939415600776dbfcbf4306b6db2d4eda725b not found: ID does not exist" Sep 17 23:40:48.706438 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.706396 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85"] Sep 17 23:40:48.708415 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.708393 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6ac6e-5696d76dcb-8sq85"] Sep 17 23:40:48.719053 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.719033 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq"] Sep 17 23:40:48.723717 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:48.723697 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6ac6e-66b668f9f4-26tvq"] Sep 17 23:40:50.090953 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:50.090917 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:40:50.286914 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:50.286871 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2842d6c8-adbb-4231-93f4-002f19cd2dd0" path="/var/lib/kubelet/pods/2842d6c8-adbb-4231-93f4-002f19cd2dd0/volumes" Sep 17 23:40:50.287259 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:50.287245 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cd59b83-c48d-4c53-9fdd-93850998ef76" path="/var/lib/kubelet/pods/8cd59b83-c48d-4c53-9fdd-93850998ef76/volumes" Sep 17 23:40:58.798324 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:40:58.798289 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:41:14.360546 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:41:14.360522 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:41:14.367853 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:41:14.367835 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:41:15.305472 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:41:15.305438 2559 scope.go:117] "RemoveContainer" containerID="528aa07e3cb429ce2cfed864dd96b5b67f4d2b71e8bfbe6bb95f5824758a1679" Sep 17 23:42:18.490474 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:42:18.490442 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:43:04.300532 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:04.300496 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:43:12.190704 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:12.190666 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:43:19.396812 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:19.396775 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:43:26.594719 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:26.594639 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:43:41.389812 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:41.389776 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:43:48.793145 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:48.793111 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:43:56.316202 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.316170 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh"] Sep 17 23:43:56.316631 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.316361 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" podUID="572db407-342d-4115-aa58-fe567fc83eb9" containerName="manager" containerID="cri-o://d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b" gracePeriod=10 Sep 17 23:43:56.564214 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.564189 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:43:56.575267 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.575207 2559 generic.go:356] "Generic (PLEG): container finished" podID="572db407-342d-4115-aa58-fe567fc83eb9" containerID="d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b" exitCode=0 Sep 17 23:43:56.575267 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.575253 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" Sep 17 23:43:56.575413 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.575286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" event={"ID":"572db407-342d-4115-aa58-fe567fc83eb9","Type":"ContainerDied","Data":"d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b"} Sep 17 23:43:56.575413 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.575337 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh" event={"ID":"572db407-342d-4115-aa58-fe567fc83eb9","Type":"ContainerDied","Data":"b5b83a8cda955e67ff67755b8450fb8370c7234f643f00cbd5fa95dfe0ba2f09"} Sep 17 23:43:56.575413 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.575359 2559 scope.go:117] "RemoveContainer" containerID="d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b" Sep 17 23:43:56.584078 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.583885 2559 scope.go:117] "RemoveContainer" containerID="6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483" Sep 17 23:43:56.593720 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.593698 2559 scope.go:117] "RemoveContainer" containerID="d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b" Sep 17 23:43:56.594078 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:43:56.594055 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b\": container with ID starting with d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b not found: ID does not exist" containerID="d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b" Sep 17 23:43:56.594183 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.594089 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b"} err="failed to get container status \"d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b\": rpc error: code = NotFound desc = could not find container \"d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b\": container with ID starting with d1fd8fd3f5a7c1be8b558806b98294ae467f131f747ccf8b2f277f5d50c48a8b not found: ID does not exist" Sep 17 23:43:56.594183 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.594112 2559 scope.go:117] "RemoveContainer" containerID="6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483" Sep 17 23:43:56.594419 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:43:56.594401 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483\": container with ID starting with 6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483 not found: ID does not exist" containerID="6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483" Sep 17 23:43:56.594497 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.594426 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483"} err="failed to get container status \"6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483\": rpc error: code = NotFound desc = could not find container \"6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483\": container with ID starting with 6328ef1f991c26ab73be2897a928384fabb462b3b4eb10eee91a8d50108e2483 not found: ID does not exist" Sep 17 23:43:56.674032 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.674006 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkbl7\" (UniqueName: \"kubernetes.io/projected/572db407-342d-4115-aa58-fe567fc83eb9-kube-api-access-gkbl7\") pod \"572db407-342d-4115-aa58-fe567fc83eb9\" (UID: \"572db407-342d-4115-aa58-fe567fc83eb9\") " Sep 17 23:43:56.674220 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.674045 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/572db407-342d-4115-aa58-fe567fc83eb9-extensions-socket-volume\") pod \"572db407-342d-4115-aa58-fe567fc83eb9\" (UID: \"572db407-342d-4115-aa58-fe567fc83eb9\") " Sep 17 23:43:56.674467 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.674443 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572db407-342d-4115-aa58-fe567fc83eb9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "572db407-342d-4115-aa58-fe567fc83eb9" (UID: "572db407-342d-4115-aa58-fe567fc83eb9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:43:56.675888 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.675867 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572db407-342d-4115-aa58-fe567fc83eb9-kube-api-access-gkbl7" (OuterVolumeSpecName: "kube-api-access-gkbl7") pod "572db407-342d-4115-aa58-fe567fc83eb9" (UID: "572db407-342d-4115-aa58-fe567fc83eb9"). InnerVolumeSpecName "kube-api-access-gkbl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:43:56.774640 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.774612 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/572db407-342d-4115-aa58-fe567fc83eb9-extensions-socket-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:43:56.774640 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.774635 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkbl7\" (UniqueName: \"kubernetes.io/projected/572db407-342d-4115-aa58-fe567fc83eb9-kube-api-access-gkbl7\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:43:56.896869 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.896841 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh"] Sep 17 23:43:56.900097 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:56.900075 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-brvgh"] Sep 17 23:43:58.286360 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:43:58.286330 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="572db407-342d-4115-aa58-fe567fc83eb9" path="/var/lib/kubelet/pods/572db407-342d-4115-aa58-fe567fc83eb9/volumes" Sep 17 23:44:56.987775 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.987695 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r"] Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988082 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d91e1518-e7f4-4c49-8fa7-f17698357a34" containerName="payload-processing" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988093 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d91e1518-e7f4-4c49-8fa7-f17698357a34" containerName="payload-processing" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988102 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="572db407-342d-4115-aa58-fe567fc83eb9" containerName="manager" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988107 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="572db407-342d-4115-aa58-fe567fc83eb9" containerName="manager" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988118 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988124 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988133 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerName="istio-proxy" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988138 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerName="istio-proxy" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988146 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerName="istio-proxy" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988151 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerName="istio-proxy" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988157 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2842d6c8-adbb-4231-93f4-002f19cd2dd0" containerName="payload-pre-processing" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988162 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2842d6c8-adbb-4231-93f4-002f19cd2dd0" containerName="payload-pre-processing" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988169 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cb90830-690f-402d-85bf-e44575d5132d" containerName="maas-api" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988173 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cb90830-690f-402d-85bf-e44575d5132d" containerName="maas-api" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988179 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988184 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988190 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e147edd-8d8f-4258-b579-74a47d20d8d2" containerName="payload-pre-processing" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988196 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e147edd-8d8f-4258-b579-74a47d20d8d2" containerName="payload-pre-processing" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988204 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd59b83-c48d-4c53-9fdd-93850998ef76" containerName="maas-api" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988209 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd59b83-c48d-4c53-9fdd-93850998ef76" containerName="maas-api" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988220 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a654e8f2-501c-4b82-8b95-77a063552922" containerName="payload-pre-processing" Sep 17 23:44:56.988233 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988225 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a654e8f2-501c-4b82-8b95-77a063552922" containerName="payload-pre-processing" Sep 17 23:44:56.988883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988289 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cb90830-690f-402d-85bf-e44575d5132d" containerName="maas-api" Sep 17 23:44:56.988883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988296 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d91e1518-e7f4-4c49-8fa7-f17698357a34" containerName="payload-processing" Sep 17 23:44:56.988883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988302 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2842d6c8-adbb-4231-93f4-002f19cd2dd0" containerName="payload-pre-processing" Sep 17 23:44:56.988883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988308 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="572db407-342d-4115-aa58-fe567fc83eb9" containerName="manager" Sep 17 23:44:56.988883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988314 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerName="istio-proxy" Sep 17 23:44:56.988883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988319 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" Sep 17 23:44:56.988883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988324 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="572db407-342d-4115-aa58-fe567fc83eb9" containerName="manager" Sep 17 23:44:56.988883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988328 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c549a62-a914-4595-8507-ca2a9000b709" containerName="istio-proxy" Sep 17 23:44:56.988883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988333 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e147edd-8d8f-4258-b579-74a47d20d8d2" containerName="payload-pre-processing" Sep 17 23:44:56.988883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988339 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a654e8f2-501c-4b82-8b95-77a063552922" containerName="payload-pre-processing" Sep 17 23:44:56.988883 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.988344 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd59b83-c48d-4c53-9fdd-93850998ef76" containerName="maas-api" Sep 17 23:44:56.991353 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.991332 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" Sep 17 23:44:56.994018 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:56.994001 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sjqbx\"" Sep 17 23:44:57.005941 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.005920 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r"] Sep 17 23:44:57.057888 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.057862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b52a2153-15ee-4d3c-93dc-09908b85b332-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z9m8r\" (UID: \"b52a2153-15ee-4d3c-93dc-09908b85b332\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" Sep 17 23:44:57.058045 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.057892 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgg5s\" (UniqueName: \"kubernetes.io/projected/b52a2153-15ee-4d3c-93dc-09908b85b332-kube-api-access-mgg5s\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z9m8r\" (UID: \"b52a2153-15ee-4d3c-93dc-09908b85b332\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" Sep 17 23:44:57.158751 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.158722 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b52a2153-15ee-4d3c-93dc-09908b85b332-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z9m8r\" (UID: \"b52a2153-15ee-4d3c-93dc-09908b85b332\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" Sep 17 23:44:57.158863 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.158755 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgg5s\" (UniqueName: \"kubernetes.io/projected/b52a2153-15ee-4d3c-93dc-09908b85b332-kube-api-access-mgg5s\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z9m8r\" (UID: \"b52a2153-15ee-4d3c-93dc-09908b85b332\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" Sep 17 23:44:57.159182 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.159164 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b52a2153-15ee-4d3c-93dc-09908b85b332-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z9m8r\" (UID: \"b52a2153-15ee-4d3c-93dc-09908b85b332\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" Sep 17 23:44:57.167106 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.167081 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgg5s\" (UniqueName: \"kubernetes.io/projected/b52a2153-15ee-4d3c-93dc-09908b85b332-kube-api-access-mgg5s\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z9m8r\" (UID: \"b52a2153-15ee-4d3c-93dc-09908b85b332\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" Sep 17 23:44:57.301041 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.300982 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" Sep 17 23:44:57.578268 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.578241 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r"] Sep 17 23:44:57.580998 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:44:57.580974 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52a2153_15ee_4d3c_93dc_09908b85b332.slice/crio-3655865d9f301f07cf7094864c68b83d46778d0b248ccd8a79cb401a9d305e01 WatchSource:0}: Error finding container 3655865d9f301f07cf7094864c68b83d46778d0b248ccd8a79cb401a9d305e01: Status 404 returned error can't find the container with id 3655865d9f301f07cf7094864c68b83d46778d0b248ccd8a79cb401a9d305e01 Sep 17 23:44:57.863530 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.863448 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" event={"ID":"b52a2153-15ee-4d3c-93dc-09908b85b332","Type":"ContainerStarted","Data":"501b37b3dd5b184fa2209ed2ea2e3dde90c44d8127a74e610eb0eacbfc092585"} Sep 17 23:44:57.863530 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.863499 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" event={"ID":"b52a2153-15ee-4d3c-93dc-09908b85b332","Type":"ContainerStarted","Data":"3655865d9f301f07cf7094864c68b83d46778d0b248ccd8a79cb401a9d305e01"} Sep 17 23:44:57.863530 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.863528 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" Sep 17 23:44:57.885103 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:44:57.884941 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" podStartSLOduration=1.884921649 podStartE2EDuration="1.884921649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:44:57.88338782 +0000 UTC m=+2326.000867297" watchObservedRunningTime="2026-09-17 23:44:57.884921649 +0000 UTC m=+2326.002401098" Sep 17 23:45:00.131976 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:00.131944 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828145-p8dvh"] Sep 17 23:45:00.132337 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:00.132318 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="572db407-342d-4115-aa58-fe567fc83eb9" containerName="manager" Sep 17 23:45:00.132337 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:00.132328 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="572db407-342d-4115-aa58-fe567fc83eb9" containerName="manager" Sep 17 23:45:00.132412 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:00.132389 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c41495-4ea9-4c22-a74d-b273f369cc31" containerName="istio-proxy" Sep 17 23:45:00.135640 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:00.135624 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-p8dvh" Sep 17 23:45:00.138118 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:00.138099 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d9szq\"" Sep 17 23:45:00.144201 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:00.144177 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828145-p8dvh"] Sep 17 23:45:00.146484 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:00.146467 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-p8dvh" Sep 17 23:45:00.415912 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:00.415822 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828145-p8dvh"] Sep 17 23:45:00.419506 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:45:00.419473 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66edc9b9_6a3d_4dd5_a041_0c21ef7b7d48.slice/crio-d84f5ade9d33507b9c25e33aa30ad082f2f106b00bb5514573e6ca61931071e7 WatchSource:0}: Error finding container d84f5ade9d33507b9c25e33aa30ad082f2f106b00bb5514573e6ca61931071e7: Status 404 returned error can't find the container with id d84f5ade9d33507b9c25e33aa30ad082f2f106b00bb5514573e6ca61931071e7 Sep 17 23:45:00.877405 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:00.877364 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-p8dvh" event={"ID":"66edc9b9-6a3d-4dd5-a041-0c21ef7b7d48","Type":"ContainerStarted","Data":"e475504654a6760cce437a2e6953d3cd025010e741c12453df6dae76a6fd79c9"} Sep 17 23:45:00.877545 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:00.877422 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-p8dvh" event={"ID":"66edc9b9-6a3d-4dd5-a041-0c21ef7b7d48","Type":"ContainerStarted","Data":"d84f5ade9d33507b9c25e33aa30ad082f2f106b00bb5514573e6ca61931071e7"} Sep 17 23:45:00.892508 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:00.892456 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-p8dvh" podStartSLOduration=0.892439902 podStartE2EDuration="892.439902ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:45:00.891430817 +0000 UTC m=+2329.008910279" watchObservedRunningTime="2026-09-17 23:45:00.892439902 +0000 UTC m=+2329.009919398" Sep 17 23:45:01.883610 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:01.883577 2559 generic.go:356] "Generic (PLEG): container finished" podID="66edc9b9-6a3d-4dd5-a041-0c21ef7b7d48" containerID="e475504654a6760cce437a2e6953d3cd025010e741c12453df6dae76a6fd79c9" exitCode=0 Sep 17 23:45:01.883996 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:01.883672 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-p8dvh" event={"ID":"66edc9b9-6a3d-4dd5-a041-0c21ef7b7d48","Type":"ContainerDied","Data":"e475504654a6760cce437a2e6953d3cd025010e741c12453df6dae76a6fd79c9"} Sep 17 23:45:03.020884 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:03.020859 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-p8dvh" Sep 17 23:45:03.894413 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:03.894381 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-p8dvh" Sep 17 23:45:03.894413 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:03.894406 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-p8dvh" event={"ID":"66edc9b9-6a3d-4dd5-a041-0c21ef7b7d48","Type":"ContainerDied","Data":"d84f5ade9d33507b9c25e33aa30ad082f2f106b00bb5514573e6ca61931071e7"} Sep 17 23:45:03.894630 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:03.894450 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d84f5ade9d33507b9c25e33aa30ad082f2f106b00bb5514573e6ca61931071e7" Sep 17 23:45:08.870180 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:08.870149 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9m8r" Sep 17 23:45:16.596868 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:16.596826 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:45:22.690960 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:22.690917 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:45:31.798634 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:31.798597 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:45:52.003529 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:45:52.003495 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:46:02.793489 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:02.793455 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:46:14.388506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:14.388474 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:46:14.399245 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:14.399225 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:46:21.101363 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:21.101323 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:46:29.902412 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:29.902328 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:46:36.702651 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.702617 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x"] Sep 17 23:46:36.703131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.703007 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66edc9b9-6a3d-4dd5-a041-0c21ef7b7d48" containerName="cleanup" Sep 17 23:46:36.703131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.703018 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="66edc9b9-6a3d-4dd5-a041-0c21ef7b7d48" containerName="cleanup" Sep 17 23:46:36.703131 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.703082 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="66edc9b9-6a3d-4dd5-a041-0c21ef7b7d48" containerName="cleanup" Sep 17 23:46:36.706291 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.706266 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.709718 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.709693 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cqmcq\"" Sep 17 23:46:36.709831 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.709693 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 23:46:36.709831 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.709691 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 17 23:46:36.709831 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.709693 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 23:46:36.710150 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.710132 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 23:46:36.717400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.717380 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x"] Sep 17 23:46:36.722949 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.722895 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv"] Sep 17 23:46:36.726109 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.726093 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:36.737197 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.737175 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv"] Sep 17 23:46:36.780864 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.780824 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a96826-e21c-4d3d-a6e9-beb28323c16c-metrics-certs\") pod \"maas-api-5d54fc9767-7zk9x\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.780864 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.780868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a96826-e21c-4d3d-a6e9-beb28323c16c-maas-api-tls\") pod \"maas-api-5d54fc9767-7zk9x\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.781104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.780893 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee-openshift-service-ca\") pod \"maas-api-86bf64cdb7-mvhvv\" (UID: \"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:36.781104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.780972 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95hsb\" (UniqueName: \"kubernetes.io/projected/f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee-kube-api-access-95hsb\") pod \"maas-api-86bf64cdb7-mvhvv\" (UID: \"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:36.781104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.781001 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee-metrics-certs\") pod \"maas-api-86bf64cdb7-mvhvv\" (UID: \"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:36.781104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.781027 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee-maas-api-tls\") pod \"maas-api-86bf64cdb7-mvhvv\" (UID: \"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:36.781104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.781042 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7pt\" (UniqueName: \"kubernetes.io/projected/b4a96826-e21c-4d3d-a6e9-beb28323c16c-kube-api-access-8c7pt\") pod \"maas-api-5d54fc9767-7zk9x\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.781287 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.781112 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a96826-e21c-4d3d-a6e9-beb28323c16c-openshift-service-ca\") pod \"maas-api-5d54fc9767-7zk9x\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.882253 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.882212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a96826-e21c-4d3d-a6e9-beb28323c16c-metrics-certs\") pod \"maas-api-5d54fc9767-7zk9x\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.882253 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.882257 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a96826-e21c-4d3d-a6e9-beb28323c16c-maas-api-tls\") pod \"maas-api-5d54fc9767-7zk9x\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.882506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.882276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee-openshift-service-ca\") pod \"maas-api-86bf64cdb7-mvhvv\" (UID: \"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:36.882506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.882398 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95hsb\" (UniqueName: \"kubernetes.io/projected/f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee-kube-api-access-95hsb\") pod \"maas-api-86bf64cdb7-mvhvv\" (UID: \"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:36.882506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.882438 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee-metrics-certs\") pod \"maas-api-86bf64cdb7-mvhvv\" (UID: \"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:36.882506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.882463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee-maas-api-tls\") pod \"maas-api-86bf64cdb7-mvhvv\" (UID: \"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:36.882506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.882477 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7pt\" (UniqueName: \"kubernetes.io/projected/b4a96826-e21c-4d3d-a6e9-beb28323c16c-kube-api-access-8c7pt\") pod \"maas-api-5d54fc9767-7zk9x\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.882760 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.882569 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a96826-e21c-4d3d-a6e9-beb28323c16c-openshift-service-ca\") pod \"maas-api-5d54fc9767-7zk9x\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.883133 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.883107 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee-openshift-service-ca\") pod \"maas-api-86bf64cdb7-mvhvv\" (UID: \"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:36.883281 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.883246 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a96826-e21c-4d3d-a6e9-beb28323c16c-openshift-service-ca\") pod \"maas-api-5d54fc9767-7zk9x\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.884950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.884921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a96826-e21c-4d3d-a6e9-beb28323c16c-maas-api-tls\") pod \"maas-api-5d54fc9767-7zk9x\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.885158 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.885137 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee-maas-api-tls\") pod \"maas-api-86bf64cdb7-mvhvv\" (UID: \"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:36.885265 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.885247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a96826-e21c-4d3d-a6e9-beb28323c16c-metrics-certs\") pod \"maas-api-5d54fc9767-7zk9x\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.885362 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.885344 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee-metrics-certs\") pod \"maas-api-86bf64cdb7-mvhvv\" (UID: \"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:36.890019 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.889998 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7pt\" (UniqueName: \"kubernetes.io/projected/b4a96826-e21c-4d3d-a6e9-beb28323c16c-kube-api-access-8c7pt\") pod \"maas-api-5d54fc9767-7zk9x\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:36.890117 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:36.890061 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95hsb\" (UniqueName: \"kubernetes.io/projected/f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee-kube-api-access-95hsb\") pod \"maas-api-86bf64cdb7-mvhvv\" (UID: \"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:37.019133 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:37.019041 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:37.037224 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:37.037193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:37.294492 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:37.294469 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x"] Sep 17 23:46:37.296630 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:46:37.296603 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a96826_e21c_4d3d_a6e9_beb28323c16c.slice/crio-32aa02e7a6d163e374872530645513a7eff69ba4edbdf09f0238dcde514b8d21 WatchSource:0}: Error finding container 32aa02e7a6d163e374872530645513a7eff69ba4edbdf09f0238dcde514b8d21: Status 404 returned error can't find the container with id 32aa02e7a6d163e374872530645513a7eff69ba4edbdf09f0238dcde514b8d21 Sep 17 23:46:37.297988 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:37.297967 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 23:46:37.328997 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:37.328968 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv"] Sep 17 23:46:37.331772 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:46:37.331743 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dc0a18_4a56_4a4f_9b93_a9db2d2815ee.slice/crio-d19dddf6f70334efb2166bfd0d53236fee6032bd545daf535f2fd1d9adc65d2b WatchSource:0}: Error finding container d19dddf6f70334efb2166bfd0d53236fee6032bd545daf535f2fd1d9adc65d2b: Status 404 returned error can't find the container with id d19dddf6f70334efb2166bfd0d53236fee6032bd545daf535f2fd1d9adc65d2b Sep 17 23:46:37.345605 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:37.345576 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" event={"ID":"b4a96826-e21c-4d3d-a6e9-beb28323c16c","Type":"ContainerStarted","Data":"32aa02e7a6d163e374872530645513a7eff69ba4edbdf09f0238dcde514b8d21"} Sep 17 23:46:37.346826 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:37.346803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" event={"ID":"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee","Type":"ContainerStarted","Data":"d19dddf6f70334efb2166bfd0d53236fee6032bd545daf535f2fd1d9adc65d2b"} Sep 17 23:46:37.396751 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:37.396718 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:46:38.354163 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:38.354123 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" event={"ID":"b4a96826-e21c-4d3d-a6e9-beb28323c16c","Type":"ContainerStarted","Data":"8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435"} Sep 17 23:46:38.354596 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:38.354191 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:38.355577 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:38.355555 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" event={"ID":"f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee","Type":"ContainerStarted","Data":"e51b87c588f98e0386aed0efaa83976c4b7212467fb76d46f978be5d4bc9c03c"} Sep 17 23:46:38.355693 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:38.355676 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:38.372461 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:38.372414 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" podStartSLOduration=1.999830775 podStartE2EDuration="2.372381988s" totalImagesPullingTime="372.551213ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:46:38.370186553 +0000 UTC m=+2426.487666020" watchObservedRunningTime="2026-09-17 23:46:38.372381988 +0000 UTC m=+2426.489861437" Sep 17 23:46:38.386731 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:38.386677 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" podStartSLOduration=1.995808665 podStartE2EDuration="2.386661006s" totalImagesPullingTime="390.852341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:46:38.386437384 +0000 UTC m=+2426.503916856" watchObservedRunningTime="2026-09-17 23:46:38.386661006 +0000 UTC m=+2426.504140456" Sep 17 23:46:44.300634 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.300598 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:46:44.367173 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.367139 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:44.367501 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.367481 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-mvhvv" Sep 17 23:46:44.437814 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.437778 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x"] Sep 17 23:46:44.438103 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.438064 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" podUID="b4a96826-e21c-4d3d-a6e9-beb28323c16c" containerName="maas-api" containerID="cri-o://8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435" gracePeriod=30 Sep 17 23:46:44.681104 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.681080 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:44.753020 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.752990 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a96826-e21c-4d3d-a6e9-beb28323c16c-maas-api-tls\") pod \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " Sep 17 23:46:44.753203 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.753029 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a96826-e21c-4d3d-a6e9-beb28323c16c-openshift-service-ca\") pod \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " Sep 17 23:46:44.753203 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.753046 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a96826-e21c-4d3d-a6e9-beb28323c16c-metrics-certs\") pod \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " Sep 17 23:46:44.753203 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.753100 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c7pt\" (UniqueName: \"kubernetes.io/projected/b4a96826-e21c-4d3d-a6e9-beb28323c16c-kube-api-access-8c7pt\") pod \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\" (UID: \"b4a96826-e21c-4d3d-a6e9-beb28323c16c\") " Sep 17 23:46:44.753509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.753481 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a96826-e21c-4d3d-a6e9-beb28323c16c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b4a96826-e21c-4d3d-a6e9-beb28323c16c" (UID: "b4a96826-e21c-4d3d-a6e9-beb28323c16c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:46:44.755300 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.755272 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a96826-e21c-4d3d-a6e9-beb28323c16c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b4a96826-e21c-4d3d-a6e9-beb28323c16c" (UID: "b4a96826-e21c-4d3d-a6e9-beb28323c16c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:46:44.755300 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.755289 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a96826-e21c-4d3d-a6e9-beb28323c16c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b4a96826-e21c-4d3d-a6e9-beb28323c16c" (UID: "b4a96826-e21c-4d3d-a6e9-beb28323c16c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:46:44.755431 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.755272 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a96826-e21c-4d3d-a6e9-beb28323c16c-kube-api-access-8c7pt" (OuterVolumeSpecName: "kube-api-access-8c7pt") pod "b4a96826-e21c-4d3d-a6e9-beb28323c16c" (UID: "b4a96826-e21c-4d3d-a6e9-beb28323c16c"). InnerVolumeSpecName "kube-api-access-8c7pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:46:44.854356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.854272 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a96826-e21c-4d3d-a6e9-beb28323c16c-maas-api-tls\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:46:44.854356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.854302 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a96826-e21c-4d3d-a6e9-beb28323c16c-openshift-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:46:44.854356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.854314 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a96826-e21c-4d3d-a6e9-beb28323c16c-metrics-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:46:44.854356 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:44.854323 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c7pt\" (UniqueName: \"kubernetes.io/projected/b4a96826-e21c-4d3d-a6e9-beb28323c16c-kube-api-access-8c7pt\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Sep 17 23:46:45.389547 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:45.389510 2559 generic.go:356] "Generic (PLEG): container finished" podID="b4a96826-e21c-4d3d-a6e9-beb28323c16c" containerID="8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435" exitCode=0 Sep 17 23:46:45.389950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:45.389556 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" Sep 17 23:46:45.389950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:45.389556 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" event={"ID":"b4a96826-e21c-4d3d-a6e9-beb28323c16c","Type":"ContainerDied","Data":"8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435"} Sep 17 23:46:45.389950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:45.389607 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x" event={"ID":"b4a96826-e21c-4d3d-a6e9-beb28323c16c","Type":"ContainerDied","Data":"32aa02e7a6d163e374872530645513a7eff69ba4edbdf09f0238dcde514b8d21"} Sep 17 23:46:45.389950 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:45.389635 2559 scope.go:117] "RemoveContainer" containerID="8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435" Sep 17 23:46:45.398971 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:45.398950 2559 scope.go:117] "RemoveContainer" containerID="8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435" Sep 17 23:46:45.399257 ip-10-0-133-222 kubenswrapper[2559]: E0917 23:46:45.399238 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435\": container with ID starting with 8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435 not found: ID does not exist" containerID="8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435" Sep 17 23:46:45.399296 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:45.399264 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435"} err="failed to get container status \"8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435\": rpc error: code = NotFound desc = could not find container \"8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435\": container with ID starting with 8e361d84e071fc9f5218b2877c831c0a6fef10e33ded9a020e05fd835da17435 not found: ID does not exist" Sep 17 23:46:45.411284 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:45.411260 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x"] Sep 17 23:46:45.414935 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:45.414887 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d54fc9767-7zk9x"] Sep 17 23:46:46.288381 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:46.288346 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4a96826-e21c-4d3d-a6e9-beb28323c16c" path="/var/lib/kubelet/pods/b4a96826-e21c-4d3d-a6e9-beb28323c16c/volumes" Sep 17 23:46:51.093956 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:51.093921 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:46:56.695509 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:46:56.695470 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:47:03.542739 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:03.542702 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rn8c6"] Sep 17 23:47:11.588874 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:11.588834 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z7jjw_a39c0513-f908-4653-8ae0-2cb0aad85b34/manager/0.log" Sep 17 23:47:11.712188 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:11.712158 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t7ncm_8c9c9334-c345-4bfd-9977-814550d49937/manager/0.log" Sep 17 23:47:12.391274 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:12.391246 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697bb5576d-qqc9p_b629004d-5aee-4574-89b8-6e63b6e968c8/copy-manifests/0.log" Sep 17 23:47:12.408270 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:12.408242 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697bb5576d-qqc9p_b629004d-5aee-4574-89b8-6e63b6e968c8/manager/0.log" Sep 17 23:47:12.750856 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:12.750827 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86bf64cdb7-mvhvv_f8dc0a18-4a56-4a4f-9b93-a9db2d2815ee/maas-api/0.log" Sep 17 23:47:13.073867 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:13.073795 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29828145-p8dvh_66edc9b9-6a3d-4dd5-a041-0c21ef7b7d48/cleanup/0.log" Sep 17 23:47:14.047024 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:14.046973 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-7jns7_a8c1ba28-60d8-42ef-a2fb-c5a3feddb06b/manager/0.log" Sep 17 23:47:14.153387 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:14.153358 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-pvmfp_bc833510-c7aa-4a82-83d1-d55269d2ed31/manager/0.log" Sep 17 23:47:14.380423 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:14.380336 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-z9m8r_b52a2153-15ee-4d3c-93dc-09908b85b332/manager/0.log" Sep 17 23:47:14.504925 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:14.504874 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-rn8c6_e570d1d6-9e91-4e60-90ea-74d24497c1d8/limitador/0.log" Sep 17 23:47:14.627101 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:14.627060 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-v6tp5_e71e542a-851c-4d11-a715-bcb7cf13b6c1/manager/0.log" Sep 17 23:47:15.178724 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:15.178691 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65f4688f79-d4v4q_94540370-de05-4b11-8010-5becb5180325/kube-auth-proxy/0.log" Sep 17 23:47:15.634225 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:15.634192 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x5r9q_c2f4d315-2130-4dc1-8c78-fca47740b436/payload-processing/0.log" Sep 17 23:47:15.743523 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:15.743489 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cd8bf6458-jhsss_0938e02b-9bf2-4f98-92b4-ee86260aef8f/router/0.log" Sep 17 23:47:16.291289 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:16.291261 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-c76f6_833c7bb0-3e22-405d-af9f-bb8cf93ef707/main/0.log" Sep 17 23:47:25.288734 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:25.288700 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-92l5f_4512e542-b8f7-4833-b2a5-9812a90f43c6/global-pull-secret-syncer/0.log" Sep 17 23:47:25.449541 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:25.449511 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9vswh_66d8dd77-971b-4a83-ab58-b49b09d40f7b/konnectivity-agent/0.log" Sep 17 23:47:25.510642 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:25.510612 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-222.ec2.internal_9ac9bbdb0f7baee4c4f7c0e8cb2517fc/haproxy/0.log" Sep 17 23:47:30.364290 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:30.364261 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-7jns7_a8c1ba28-60d8-42ef-a2fb-c5a3feddb06b/manager/0.log" Sep 17 23:47:30.391958 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:30.391930 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-pvmfp_bc833510-c7aa-4a82-83d1-d55269d2ed31/manager/0.log" Sep 17 23:47:30.465172 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:30.465131 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-z9m8r_b52a2153-15ee-4d3c-93dc-09908b85b332/manager/0.log" Sep 17 23:47:30.486390 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:30.486359 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-rn8c6_e570d1d6-9e91-4e60-90ea-74d24497c1d8/limitador/0.log" Sep 17 23:47:30.575779 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:30.575751 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-v6tp5_e71e542a-851c-4d11-a715-bcb7cf13b6c1/manager/0.log" Sep 17 23:47:32.268919 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:32.268869 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-r2vwj_b1a856fa-a9da-41e6-8a80-da9ddc2d5cb5/cluster-monitoring-operator/0.log" Sep 17 23:47:32.505400 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:32.505357 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-njz52_6351ced2-25b1-44c2-9ceb-17294d942b5c/node-exporter/0.log" Sep 17 23:47:32.526781 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:32.526757 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-njz52_6351ced2-25b1-44c2-9ceb-17294d942b5c/kube-rbac-proxy/0.log" Sep 17 23:47:32.548297 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:32.548281 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-njz52_6351ced2-25b1-44c2-9ceb-17294d942b5c/init-textfile/0.log" Sep 17 23:47:34.606242 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:34.606214 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-dp5fl_8b5569f3-268e-4bd8-b7dc-6f5f0102f08c/networking-console-plugin/0.log" Sep 17 23:47:35.295571 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.295536 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/1.log" Sep 17 23:47:35.305582 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.305551 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kfrv8_c69cad18-c083-405a-aed3-91740cc79fd4/console-operator/2.log" Sep 17 23:47:35.932550 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.932509 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz"] Sep 17 23:47:35.933127 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.933107 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a96826-e21c-4d3d-a6e9-beb28323c16c" containerName="maas-api" Sep 17 23:47:35.933212 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.933130 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a96826-e21c-4d3d-a6e9-beb28323c16c" containerName="maas-api" Sep 17 23:47:35.933265 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.933219 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a96826-e21c-4d3d-a6e9-beb28323c16c" containerName="maas-api" Sep 17 23:47:35.937535 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.937514 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:35.939857 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.939839 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7zwb4\"/\"openshift-service-ca.crt\"" Sep 17 23:47:35.939962 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.939943 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7zwb4\"/\"kube-root-ca.crt\"" Sep 17 23:47:35.940782 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.940766 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7zwb4\"/\"default-dockercfg-xtqbw\"" Sep 17 23:47:35.943065 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.943036 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz"] Sep 17 23:47:35.979391 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.979363 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7297320-31eb-4f66-a097-89f2b5768ead-proc\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:35.979512 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.979407 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7297320-31eb-4f66-a097-89f2b5768ead-sys\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:35.979512 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.979432 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7297320-31eb-4f66-a097-89f2b5768ead-lib-modules\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:35.979512 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.979491 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ls2r\" (UniqueName: \"kubernetes.io/projected/b7297320-31eb-4f66-a097-89f2b5768ead-kube-api-access-6ls2r\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:35.979625 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:35.979537 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7297320-31eb-4f66-a097-89f2b5768ead-podres\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:36.026141 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.026120 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-kzt9c_46c568d6-e39e-4f20-89a9-376e4b667e20/download-server/0.log" Sep 17 23:47:36.080931 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.080886 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7297320-31eb-4f66-a097-89f2b5768ead-podres\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:36.081091 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.080950 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7297320-31eb-4f66-a097-89f2b5768ead-proc\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:36.081091 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.080980 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7297320-31eb-4f66-a097-89f2b5768ead-sys\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:36.081091 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.080995 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7297320-31eb-4f66-a097-89f2b5768ead-lib-modules\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:36.081091 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.081029 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7297320-31eb-4f66-a097-89f2b5768ead-podres\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:36.081091 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.081040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ls2r\" (UniqueName: \"kubernetes.io/projected/b7297320-31eb-4f66-a097-89f2b5768ead-kube-api-access-6ls2r\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:36.081091 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.081071 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7297320-31eb-4f66-a097-89f2b5768ead-proc\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:36.081312 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.081071 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7297320-31eb-4f66-a097-89f2b5768ead-sys\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:36.081312 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.081157 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7297320-31eb-4f66-a097-89f2b5768ead-lib-modules\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:36.088816 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.088788 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ls2r\" (UniqueName: \"kubernetes.io/projected/b7297320-31eb-4f66-a097-89f2b5768ead-kube-api-access-6ls2r\") pod \"perf-node-gather-daemonset-287lz\" (UID: \"b7297320-31eb-4f66-a097-89f2b5768ead\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:36.248257 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.248233 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:36.523122 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.523095 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz"] Sep 17 23:47:36.526202 ip-10-0-133-222 kubenswrapper[2559]: W0917 23:47:36.526175 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb7297320_31eb_4f66_a097_89f2b5768ead.slice/crio-2c0b6eb6889dee61bc7c8b734cd12ff1d86ca89d7a4311e0af4b03486260f385 WatchSource:0}: Error finding container 2c0b6eb6889dee61bc7c8b734cd12ff1d86ca89d7a4311e0af4b03486260f385: Status 404 returned error can't find the container with id 2c0b6eb6889dee61bc7c8b734cd12ff1d86ca89d7a4311e0af4b03486260f385 Sep 17 23:47:36.641657 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.641626 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" event={"ID":"b7297320-31eb-4f66-a097-89f2b5768ead","Type":"ContainerStarted","Data":"2c0b6eb6889dee61bc7c8b734cd12ff1d86ca89d7a4311e0af4b03486260f385"} Sep 17 23:47:36.763403 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:36.763380 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-sx67h_97e32c44-0147-4b12-ac41-5f6643cbd98f/volume-data-source-validator/0.log" Sep 17 23:47:37.647980 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:37.647944 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" event={"ID":"b7297320-31eb-4f66-a097-89f2b5768ead","Type":"ContainerStarted","Data":"9bc279108e38b5fba7addbbbd6fee571f4143b577745e2dbfba987f671bdf6db"} Sep 17 23:47:37.648377 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:37.648034 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:37.664289 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:37.664243 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" podStartSLOduration=2.6642277659999998 podStartE2EDuration="2.664227766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:47:37.66261194 +0000 UTC m=+2485.780091414" watchObservedRunningTime="2026-09-17 23:47:37.664227766 +0000 UTC m=+2485.781707214" Sep 17 23:47:37.861828 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:37.861776 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2fdkb_5b7ac966-8594-4a9f-aa3f-85f12087b2a3/dns/0.log" Sep 17 23:47:37.880592 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:37.880558 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2fdkb_5b7ac966-8594-4a9f-aa3f-85f12087b2a3/kube-rbac-proxy/0.log" Sep 17 23:47:37.985021 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:37.984994 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fxvl4_d6070593-232f-4a38-b8e2-3380c4abe49e/dns-node-resolver/0.log" Sep 17 23:47:38.609051 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:38.609019 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l5s9l_064ea6f5-bd51-42ec-825e-25f7ea4a6723/node-ca/0.log" Sep 17 23:47:40.092700 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:40.092673 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65f4688f79-d4v4q_94540370-de05-4b11-8010-5becb5180325/kube-auth-proxy/0.log" Sep 17 23:47:40.224414 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:40.224384 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x5r9q_c2f4d315-2130-4dc1-8c78-fca47740b436/payload-processing/0.log" Sep 17 23:47:40.246226 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:40.246201 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cd8bf6458-jhsss_0938e02b-9bf2-4f98-92b4-ee86260aef8f/router/0.log" Sep 17 23:47:41.120944 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:41.120889 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6xlrg_d16d7679-5919-42cb-b4c7-379fadd35925/serve-healthcheck-canary/0.log" Sep 17 23:47:41.643813 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:41.643767 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-gw259_555eaa00-e5cb-49e7-adbb-18ccfa293242/insights-operator/0.log" Sep 17 23:47:41.647669 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:41.647644 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-gw259_555eaa00-e5cb-49e7-adbb-18ccfa293242/insights-operator/1.log" Sep 17 23:47:41.734380 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:41.734357 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhnn4_22411ad6-db7e-412f-9088-4da16631a9cf/kube-rbac-proxy/0.log" Sep 17 23:47:41.756459 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:41.756432 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhnn4_22411ad6-db7e-412f-9088-4da16631a9cf/exporter/0.log" Sep 17 23:47:41.777106 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:41.777083 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhnn4_22411ad6-db7e-412f-9088-4da16631a9cf/extractor/0.log" Sep 17 23:47:43.662290 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:43.662265 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-287lz" Sep 17 23:47:43.881387 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:43.881360 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z7jjw_a39c0513-f908-4653-8ae0-2cb0aad85b34/manager/0.log" Sep 17 23:47:44.347485 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:44.347459 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t7ncm_8c9c9334-c345-4bfd-9977-814550d49937/manager/0.log" Sep 17 23:47:44.773409 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:44.773374 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697bb5576d-qqc9p_b629004d-5aee-4574-89b8-6e63b6e968c8/manager/0.log" Sep 17 23:47:44.794061 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:44.794042 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697bb5576d-qqc9p_b629004d-5aee-4574-89b8-6e63b6e968c8/copy-manifests/0.log" Sep 17 23:47:47.217266 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:47.217233 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65f4688f79-d4v4q_94540370-de05-4b11-8010-5becb5180325/kube-auth-proxy/0.log" Sep 17 23:47:47.320506 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:47.320479 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x5r9q_c2f4d315-2130-4dc1-8c78-fca47740b436/payload-processing/0.log" Sep 17 23:47:47.341481 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:47.341427 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cd8bf6458-jhsss_0938e02b-9bf2-4f98-92b4-ee86260aef8f/router/0.log" Sep 17 23:47:49.491358 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:49.491320 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65f4688f79-d4v4q_94540370-de05-4b11-8010-5becb5180325/kube-auth-proxy/0.log" Sep 17 23:47:49.604696 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:49.604665 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x5r9q_c2f4d315-2130-4dc1-8c78-fca47740b436/payload-processing/0.log" Sep 17 23:47:49.620347 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:49.620320 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cd8bf6458-jhsss_0938e02b-9bf2-4f98-92b4-ee86260aef8f/router/0.log" Sep 17 23:47:51.497370 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:51.497334 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-r8t9z_252f319b-a647-4f72-915c-2a4df27d83fe/migrator/0.log" Sep 17 23:47:51.515246 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:51.515221 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-r8t9z_252f319b-a647-4f72-915c-2a4df27d83fe/graceful-termination/0.log" Sep 17 23:47:51.871170 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:51.871143 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-4gmns_a9479c5a-db24-4df7-9402-b6dffa701fa4/kube-storage-version-migrator-operator/1.log" Sep 17 23:47:51.872440 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:51.872418 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-4gmns_a9479c5a-db24-4df7-9402-b6dffa701fa4/kube-storage-version-migrator-operator/0.log" Sep 17 23:47:53.217737 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:53.217709 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx2wl_4cd6491b-4078-4652-98cc-31a2b75c6911/kube-multus-additional-cni-plugins/0.log" Sep 17 23:47:53.238283 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:53.238258 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx2wl_4cd6491b-4078-4652-98cc-31a2b75c6911/egress-router-binary-copy/0.log" Sep 17 23:47:53.259193 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:53.259160 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx2wl_4cd6491b-4078-4652-98cc-31a2b75c6911/cni-plugins/0.log" Sep 17 23:47:53.281408 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:53.281381 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx2wl_4cd6491b-4078-4652-98cc-31a2b75c6911/bond-cni-plugin/0.log" Sep 17 23:47:53.312322 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:53.312295 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx2wl_4cd6491b-4078-4652-98cc-31a2b75c6911/routeoverride-cni/0.log" Sep 17 23:47:53.334748 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:53.334724 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx2wl_4cd6491b-4078-4652-98cc-31a2b75c6911/whereabouts-cni-bincopy/0.log" Sep 17 23:47:53.357598 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:53.357579 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx2wl_4cd6491b-4078-4652-98cc-31a2b75c6911/whereabouts-cni/0.log" Sep 17 23:47:53.485070 ip-10-0-133-222 kubenswrapper[2559]: I0917 23:47:53.484983 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nvk6c_99ae3825-31b7-4c22-bf96-d6ee2c98f2c9/kube-multus/0.log"