Jul 29 20:45:29.859175 ip-10-0-137-160 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 29 20:45:29.859188 ip-10-0-137-160 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 29 20:45:29.859197 ip-10-0-137-160 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 29 20:45:29.859635 ip-10-0-137-160 systemd[1]: Failed to start Kubernetes Kubelet. Jul 29 20:45:40.037445 ip-10-0-137-160 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 29 20:45:40.037459 ip-10-0-137-160 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot bb9a026a5ddd4e3ebea48bb80ed1b076 -- Jul 29 20:47:31.752354 ip-10-0-137-160 systemd[1]: Starting Kubernetes Kubelet... Jul 29 20:47:32.182285 ip-10-0-137-160 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 20:47:32.182285 ip-10-0-137-160 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 20:47:32.182285 ip-10-0-137-160 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 20:47:32.182285 ip-10-0-137-160 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 20:47:32.182285 ip-10-0-137-160 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 20:47:32.185084 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.184979 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 20:47:32.190885 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190870 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 20:47:32.190885 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190885 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190889 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190893 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190896 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190901 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190904 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190906 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190909 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190912 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190915 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190918 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190920 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190923 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190926 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190928 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190931 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190934 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190936 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190939 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190941 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 20:47:32.190950 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190944 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190947 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190950 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190953 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190956 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190959 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190962 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190965 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190968 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190972 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190976 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190979 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190982 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190985 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190988 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190991 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190993 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190996 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.190999 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 20:47:32.191437 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191001 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191004 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191006 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191010 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191012 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191015 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191017 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191035 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191039 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191044 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191048 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191054 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191058 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191062 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191065 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191068 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191071 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191074 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191077 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 20:47:32.191894 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191079 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191082 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191084 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191087 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191090 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191092 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191095 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191098 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191100 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191103 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191106 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191109 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191111 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191114 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191117 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191120 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191122 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191124 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191129 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191132 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 20:47:32.192498 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191135 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191137 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191140 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191142 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191145 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191148 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191150 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191153 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191156 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191159 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191161 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191164 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191166 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191169 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191171 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191174 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191176 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191179 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191181 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.191184 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 20:47:32.192978 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192637 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192646 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192650 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192653 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192656 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192666 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192669 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192672 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192675 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192678 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192681 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192684 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192686 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192689 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192691 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192694 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192697 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192699 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192702 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192705 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 20:47:32.193479 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192708 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192710 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192714 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192718 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192721 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192724 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192727 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192730 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192733 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192736 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192738 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192741 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192743 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192746 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192748 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192751 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192755 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192757 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192760 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 20:47:32.193987 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192763 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192765 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192769 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192771 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192774 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192776 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192779 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192781 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192784 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192786 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192789 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192791 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192794 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192797 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192799 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192802 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192804 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192807 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192809 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192813 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 20:47:32.194472 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192817 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192819 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192822 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192825 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192827 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192830 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192832 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192835 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192837 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192840 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192842 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192845 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192848 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192850 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192853 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192856 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192858 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192862 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192864 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192867 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 20:47:32.194965 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192869 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192871 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192874 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192879 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192881 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192884 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192887 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192889 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192892 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192894 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192897 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192900 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192902 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192905 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192908 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192910 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192913 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192915 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192918 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.192921 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 20:47:32.195468 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193296 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193301 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193303 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193306 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193309 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193312 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193315 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193318 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193321 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193323 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193326 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193328 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193331 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193333 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193336 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193339 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193342 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193345 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193347 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193349 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 20:47:32.195954 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193352 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193355 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193358 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193360 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193362 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193365 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193367 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193370 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193372 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193374 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193377 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193379 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193382 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193385 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193387 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193391 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193394 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193398 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193401 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 20:47:32.196462 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193405 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193407 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193410 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193413 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193416 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193418 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193421 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193424 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193427 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193430 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193433 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193435 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193437 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193440 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193443 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193445 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193448 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193450 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193453 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193455 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 20:47:32.196913 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193458 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193460 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193463 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193465 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193467 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193470 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193475 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193478 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193481 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193483 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193486 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193491 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193494 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193496 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193499 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193502 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193504 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193506 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193509 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 20:47:32.197439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193511 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193514 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193517 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193520 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193522 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193525 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193529 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193532 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193534 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193537 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193540 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193542 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193544 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193547 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193549 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193552 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193554 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193557 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193559 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 20:47:32.197899 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193562 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.193565 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193644 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193654 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193660 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193664 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193671 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193674 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193678 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193682 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193685 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193688 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193692 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193695 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193698 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193701 2582 flags.go:64] FLAG: --cgroup-root="" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193703 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193706 2582 flags.go:64] FLAG: --client-ca-file="" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193709 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193712 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193716 2582 flags.go:64] FLAG: --cluster-domain="" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193719 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193722 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193725 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 20:47:32.198394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193728 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193732 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193735 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193738 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193741 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193744 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193747 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193749 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193752 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193755 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193759 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193762 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193765 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193767 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193770 2582 flags.go:64] FLAG: --enable-server="true" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193774 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193778 2582 flags.go:64] FLAG: --event-burst="100" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193793 2582 flags.go:64] FLAG: --event-qps="50" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193797 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193801 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193804 2582 flags.go:64] FLAG: --eviction-hard="" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193808 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193813 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193816 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193819 2582 flags.go:64] FLAG: --eviction-soft="" Jul 29 20:47:32.198982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193821 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193824 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193827 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193830 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193833 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193836 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193839 2582 flags.go:64] FLAG: --feature-gates="" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193842 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193845 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193848 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193851 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193854 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193857 2582 flags.go:64] FLAG: --help="false" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193859 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193862 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193865 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193868 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193872 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193875 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193877 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193880 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193883 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193885 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 20:47:32.199590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193889 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193892 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193895 2582 flags.go:64] FLAG: --kube-reserved="" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193897 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193900 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193903 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193907 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193909 2582 flags.go:64] FLAG: --lock-file="" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193912 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193915 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193918 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193923 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193926 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193929 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193931 2582 flags.go:64] FLAG: --logging-format="text" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193934 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193937 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193940 2582 flags.go:64] FLAG: --manifest-url="" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193948 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193952 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193955 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193959 2582 flags.go:64] FLAG: --max-pods="110" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193962 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193965 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193967 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 20:47:32.200178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193971 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193973 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193976 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193979 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193986 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193989 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193992 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193995 2582 flags.go:64] FLAG: --pod-cidr="" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.193998 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194004 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194007 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194010 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194013 2582 flags.go:64] FLAG: --port="10250" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194016 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194035 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ce01504c90aa6bad" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194038 2582 flags.go:64] FLAG: --qos-reserved="" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194041 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194044 2582 flags.go:64] FLAG: --register-node="true" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194047 2582 flags.go:64] FLAG: --register-with-taints="" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194051 2582 flags.go:64] FLAG: --registry-burst="10" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194054 2582 flags.go:64] FLAG: --registry-qps="5" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194057 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194060 2582 flags.go:64] FLAG: --reserved-memory="" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194063 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194066 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 20:47:32.200823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194072 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194074 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194077 2582 flags.go:64] FLAG: --runonce="false" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194080 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194083 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194086 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194088 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194091 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194094 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194097 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194100 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194103 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194106 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194108 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194111 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194114 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194117 2582 flags.go:64] FLAG: --system-cgroups="" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194120 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194125 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194128 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194131 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194137 2582 flags.go:64] FLAG: --tls-min-version="" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194139 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194142 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194145 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 20:47:32.201458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194148 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194151 2582 flags.go:64] FLAG: --v="2" Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194154 2582 flags.go:64] FLAG: --version="false" Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194158 2582 flags.go:64] FLAG: --vmodule="" Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194162 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.194165 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194276 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194283 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194286 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194290 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194293 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194296 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194299 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194302 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194304 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194307 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194310 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194312 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194315 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194317 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194320 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 20:47:32.202084 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194323 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194325 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194328 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194333 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194336 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194339 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194341 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194345 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194348 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194350 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194353 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194355 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194358 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194361 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194363 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194365 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194368 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194371 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194374 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194377 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 20:47:32.202600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194380 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194382 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194384 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194387 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194390 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194392 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194395 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194397 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194400 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194403 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194405 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194408 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194410 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194413 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194415 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194418 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194421 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194423 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194426 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194429 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 20:47:32.203111 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194432 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194435 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194437 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194440 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194443 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194445 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194448 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194450 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194453 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194456 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194459 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194462 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194464 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194467 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194470 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194472 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194475 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194478 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194480 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 20:47:32.203592 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194484 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194488 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194492 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194495 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194498 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194501 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194504 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194507 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194510 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194512 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194515 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194518 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194523 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194525 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194528 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194530 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194533 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194536 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194538 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 20:47:32.204060 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194541 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 20:47:32.204525 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194543 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 20:47:32.204525 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194546 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 20:47:32.204525 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194548 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 20:47:32.204525 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194552 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 20:47:32.204525 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.194554 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 20:47:32.204525 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.195445 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 20:47:32.214662 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.214643 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 20:47:32.214662 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.214661 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.214679 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.214685 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214741 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214746 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214749 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214752 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214755 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214758 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214761 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214763 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214766 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214769 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214772 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 20:47:32.214768 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214775 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214778 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214781 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214784 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214788 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214790 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214795 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214799 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214802 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214805 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214808 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214811 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214814 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214816 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214819 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214822 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214824 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214827 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214830 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 20:47:32.215247 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214832 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214835 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214837 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214840 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214842 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214845 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214848 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214850 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214853 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214855 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214858 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214860 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214863 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214866 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214868 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214872 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214875 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214878 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214880 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214883 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 20:47:32.215710 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214885 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214888 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214890 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214893 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214895 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214898 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214900 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214903 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214905 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214908 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214910 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214913 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214916 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214918 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214921 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214923 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214926 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214928 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214932 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214936 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 20:47:32.216214 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214938 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214941 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214944 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214946 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214949 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214951 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214954 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214958 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214961 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214965 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214969 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214972 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214975 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214977 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214980 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214983 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214985 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214988 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214990 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214993 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 20:47:32.216697 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214995 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.214998 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215000 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215003 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215005 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215008 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215010 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215013 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215015 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.215037 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215173 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215178 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215181 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215184 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215186 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 20:47:32.217194 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215189 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215192 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215195 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215197 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215200 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215203 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215206 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215209 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215211 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215214 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215217 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215219 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215222 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215224 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215227 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215229 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215231 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215234 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215236 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215239 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 20:47:32.217563 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215241 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215244 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215246 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215249 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215252 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215254 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215257 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215259 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215262 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215265 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215267 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215270 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215272 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215275 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215277 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215280 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215283 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215285 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215288 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215293 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 20:47:32.218074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215296 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215299 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215301 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215304 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215306 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215309 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215311 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215314 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215317 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215319 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215322 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215324 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215327 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215330 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215332 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215335 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215337 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215340 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215342 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215345 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 20:47:32.218554 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215348 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215350 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215353 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215356 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215360 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215363 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215366 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215369 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215373 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215377 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215380 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215383 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215387 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215391 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215394 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215397 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215400 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215402 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215405 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 20:47:32.219054 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215407 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215410 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215412 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215415 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215417 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215420 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215422 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215424 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215427 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215429 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215432 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215434 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215437 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215439 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:32.215442 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 20:47:32.219565 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.215446 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 20:47:32.219955 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.215545 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 20:47:32.219955 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.217829 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 20:47:32.219955 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.219312 2582 server.go:1019] "Starting client certificate rotation" Jul 29 20:47:32.219955 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.219413 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 20:47:32.219955 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.219446 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 20:47:32.221885 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.221871 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 20:47:32.225113 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.225057 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 20:47:32.232832 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.232818 2582 log.go:25] "Validated CRI v1 runtime API" Jul 29 20:47:32.235005 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.234988 2582 log.go:25] "Validated CRI v1 image API" Jul 29 20:47:32.235841 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.235818 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 20:47:32.238689 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.238661 2582 fs.go:135] Filesystem UUIDs: map[25ac797a-486d-461b-9256-06342694d162:/dev/nvme0n1p3 4c28bf18-1382-4cbf-a8ac-e1c3e63836c7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 29 20:47:32.238771 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.238688 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 20:47:32.243893 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.243775 2582 manager.go:216] Machine: {Timestamp:2026-07-29 20:47:32.242416595 +0000 UTC m=+0.366006428 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100767 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b9e808aed684ca167dfc46d1c6ca7 SystemUUID:ec2b9e80-8aed-684c-a167-dfc46d1c6ca7 BootID:bb9a026a-5ddd-4e3e-bea4-8bb80ed1b076 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:06:5a:1c:4a:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:06:5a:1c:4a:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:7f:3c:a5:4b:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 20:47:32.243893 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.243881 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 20:47:32.244076 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.244045 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 20:47:32.247669 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.247641 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 20:47:32.247809 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.247673 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-160.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 20:47:32.249100 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.249090 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 20:47:32.249140 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.249103 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 20:47:32.249140 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.249110 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 20:47:32.249988 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.249978 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 20:47:32.251053 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.251043 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 20:47:32.253103 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.253092 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 29 20:47:32.253217 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.253207 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 20:47:32.253247 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.253239 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 29 20:47:32.253274 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.253249 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 20:47:32.253274 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.253259 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 20:47:32.253274 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.253268 2582 kubelet.go:398] "Adding apiserver pod source" Jul 29 20:47:32.253365 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.253277 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 20:47:32.253761 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.253746 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 20:47:32.253807 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.253768 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 20:47:32.256274 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.256254 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 20:47:32.256862 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.256844 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 20:47:32.259005 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.258988 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 20:47:32.259074 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.259037 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 20:47:32.260879 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.260867 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 20:47:32.260925 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.260886 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 20:47:32.260925 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.260900 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 20:47:32.260925 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.260910 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 20:47:32.260925 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.260921 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 20:47:32.260925 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.260926 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 20:47:32.261083 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.260934 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 20:47:32.261083 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.260939 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 20:47:32.261083 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.260950 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 20:47:32.261083 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.260955 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 20:47:32.261083 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.260964 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 20:47:32.261083 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.260969 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 20:47:32.261702 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.261691 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 20:47:32.261702 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.261701 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 20:47:32.262491 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.262465 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 20:47:32.262576 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.262492 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 20:47:32.265124 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.265111 2582 server.go:1268] "Started kubelet" Jul 29 20:47:32.265712 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.265575 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 20:47:32.265712 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.265554 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 20:47:32.265853 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.265829 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 20:47:32.266363 ip-10-0-137-160 systemd[1]: Started Kubernetes Kubelet. Jul 29 20:47:32.268121 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.268103 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 20:47:32.269503 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.269486 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 29 20:47:32.271402 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.271381 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 20:47:32.274046 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.274009 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fxmmt" Jul 29 20:47:32.275219 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.275202 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-160.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 20:47:32.276222 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.276201 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 20:47:32.276222 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.276207 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 20:47:32.276410 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.276396 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:32.276728 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.275231 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-160.ec2.internal.18c6ddb07624c561 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-160.ec2.internal,UID:ip-10-0-137-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-160.ec2.internal,},FirstTimestamp:2026-07-29 20:47:32.265084257 +0000 UTC m=+0.388674090,LastTimestamp:2026-07-29 20:47:32.265084257 +0000 UTC m=+0.388674090,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-160.ec2.internal,}" Jul 29 20:47:32.276856 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.276841 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 20:47:32.276856 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.276844 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 20:47:32.276963 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.276866 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 20:47:32.276963 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.276960 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 29 20:47:32.277071 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.276969 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 29 20:47:32.277177 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.277111 2582 factory.go:55] Registering systemd factory Jul 29 20:47:32.277177 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.277141 2582 factory.go:223] Registration of the systemd container factory successfully Jul 29 20:47:32.277384 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.277368 2582 factory.go:153] Registering CRI-O factory Jul 29 20:47:32.277458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.277387 2582 factory.go:223] Registration of the crio container factory successfully Jul 29 20:47:32.277458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.277440 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 20:47:32.277541 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.277459 2582 factory.go:103] Registering Raw factory Jul 29 20:47:32.277541 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.277473 2582 manager.go:1196] Started watching for new ooms in manager Jul 29 20:47:32.277988 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.277853 2582 manager.go:349] Starting recovery of all containers Jul 29 20:47:32.280162 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.280137 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fxmmt" Jul 29 20:47:32.283920 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.283894 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 20:47:32.285532 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.285501 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 20:47:32.286368 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.286348 2582 manager.go:354] Recovery completed Jul 29 20:47:32.291342 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.291330 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:47:32.293533 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.293519 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:47:32.293577 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.293551 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:47:32.293577 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.293565 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:47:32.294058 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.294042 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 20:47:32.294058 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.294054 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 20:47:32.294153 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.294066 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:47:32.294153 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.294071 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 29 20:47:32.295552 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.295536 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:47:32.295611 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.295570 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:47:32.295611 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.295583 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:47:32.295678 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.295473 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-160.ec2.internal.18c6ddb077d6e14a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-160.ec2.internal,UID:ip-10-0-137-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-160.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-160.ec2.internal,},FirstTimestamp:2026-07-29 20:47:32.293534026 +0000 UTC m=+0.417123862,LastTimestamp:2026-07-29 20:47:32.293534026 +0000 UTC m=+0.417123862,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-160.ec2.internal,}" Jul 29 20:47:32.296221 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.296209 2582 policy_none.go:49] "None policy: Start" Jul 29 20:47:32.296255 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.296226 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 20:47:32.296255 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.296236 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 20:47:32.297338 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.297327 2582 policy_none.go:47] "Start" Jul 29 20:47:32.335578 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.335561 2582 manager.go:337] "Starting Device Plugin manager" Jul 29 20:47:32.335654 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.335633 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 20:47:32.335654 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.335643 2582 server.go:87] "Starting device plugin registration server" Jul 29 20:47:32.335898 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.335884 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 20:47:32.335933 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.335907 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 20:47:32.336001 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.335984 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 20:47:32.336105 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.336090 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 20:47:32.336105 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.336100 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 20:47:32.336616 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.336594 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 20:47:32.336681 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.336651 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:32.436399 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.436326 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:47:32.437609 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.437314 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:47:32.437609 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.437347 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:47:32.437609 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.437358 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:47:32.437609 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.437383 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.445987 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.445968 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.446090 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.445994 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-160.ec2.internal\": node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:32.461258 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.461231 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:32.485138 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.485108 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 20:47:32.486444 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.486420 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 20:47:32.486540 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.486473 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 20:47:32.486540 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.486501 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 20:47:32.486622 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.486544 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 20:47:32.489047 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.489007 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 20:47:32.561577 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.561539 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:32.587767 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.586972 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-160.ec2.internal"] Jul 29 20:47:32.587834 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.587813 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:47:32.589042 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.589011 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:47:32.589139 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.589054 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:47:32.589139 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.589064 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:47:32.590355 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.590343 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:47:32.590466 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.590453 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.590509 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.590498 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:47:32.591316 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.591298 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:47:32.591387 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.591321 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:47:32.591387 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.591337 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:47:32.591462 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.591394 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:47:32.591462 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.591421 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:47:32.591462 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.591431 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:47:32.592706 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.592693 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.592754 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.592715 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:47:32.593464 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.593450 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:47:32.593540 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.593470 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:47:32.593540 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.593480 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:47:32.615753 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.615733 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-160.ec2.internal\" not found" node="ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.620245 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.620229 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-160.ec2.internal\" not found" node="ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.662053 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.662013 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:32.762687 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.762621 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:32.779101 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.779013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21c1930d16639ed349b597b7c5397dfb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal\" (UID: \"21c1930d16639ed349b597b7c5397dfb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.779162 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.779116 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21c1930d16639ed349b597b7c5397dfb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal\" (UID: \"21c1930d16639ed349b597b7c5397dfb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.779162 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.779133 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8e6a9ef8e0380c11ab0ad572fa464511-config\") pod \"kube-apiserver-proxy-ip-10-0-137-160.ec2.internal\" (UID: \"8e6a9ef8e0380c11ab0ad572fa464511\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.863199 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.863160 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:32.879517 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.879492 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8e6a9ef8e0380c11ab0ad572fa464511-config\") pod \"kube-apiserver-proxy-ip-10-0-137-160.ec2.internal\" (UID: \"8e6a9ef8e0380c11ab0ad572fa464511\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.879567 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.879522 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21c1930d16639ed349b597b7c5397dfb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal\" (UID: \"21c1930d16639ed349b597b7c5397dfb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.879567 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.879544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21c1930d16639ed349b597b7c5397dfb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal\" (UID: \"21c1930d16639ed349b597b7c5397dfb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.879630 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.879592 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21c1930d16639ed349b597b7c5397dfb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal\" (UID: \"21c1930d16639ed349b597b7c5397dfb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.879630 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.879605 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21c1930d16639ed349b597b7c5397dfb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal\" (UID: \"21c1930d16639ed349b597b7c5397dfb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.879691 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.879592 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8e6a9ef8e0380c11ab0ad572fa464511-config\") pod \"kube-apiserver-proxy-ip-10-0-137-160.ec2.internal\" (UID: \"8e6a9ef8e0380c11ab0ad572fa464511\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.918685 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.918645 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.923807 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:32.923790 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.ec2.internal" Jul 29 20:47:32.964113 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:32.964077 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:33.064637 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:33.064552 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:33.165158 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:33.165128 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:33.218644 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.218615 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 20:47:33.219277 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.218758 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 20:47:33.266167 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:33.266133 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:33.285494 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.285451 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 20:42:32 +0000 UTC" deadline="2028-03-26 06:09:31.540787285 +0000 UTC" Jul 29 20:47:33.285494 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.285487 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14529h21m58.255303617s" Jul 29 20:47:33.366414 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:33.366333 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:33.466888 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:33.466865 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:33.474074 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:33.474038 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6a9ef8e0380c11ab0ad572fa464511.slice/crio-96a720a3355d102d9d052ff44c3eed1e227c9e15267d77bda4d7c626d49c9ce7 WatchSource:0}: Error finding container 96a720a3355d102d9d052ff44c3eed1e227c9e15267d77bda4d7c626d49c9ce7: Status 404 returned error can't find the container with id 96a720a3355d102d9d052ff44c3eed1e227c9e15267d77bda4d7c626d49c9ce7 Jul 29 20:47:33.474307 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:33.474293 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c1930d16639ed349b597b7c5397dfb.slice/crio-11ac1af704179761c76c47cd4489a438bc633e35c9f3d14c7916f030371d7344 WatchSource:0}: Error finding container 11ac1af704179761c76c47cd4489a438bc633e35c9f3d14c7916f030371d7344: Status 404 returned error can't find the container with id 11ac1af704179761c76c47cd4489a438bc633e35c9f3d14c7916f030371d7344 Jul 29 20:47:33.478525 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.478499 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 20:47:33.490180 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.490142 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.ec2.internal" event={"ID":"8e6a9ef8e0380c11ab0ad572fa464511","Type":"ContainerStarted","Data":"96a720a3355d102d9d052ff44c3eed1e227c9e15267d77bda4d7c626d49c9ce7"} Jul 29 20:47:33.491211 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.491190 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" event={"ID":"21c1930d16639ed349b597b7c5397dfb","Type":"ContainerStarted","Data":"11ac1af704179761c76c47cd4489a438bc633e35c9f3d14c7916f030371d7344"} Jul 29 20:47:33.506820 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.506791 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 20:47:33.567966 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:33.567937 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:33.668509 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:33.668425 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:33.698505 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.698478 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 20:47:33.768828 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:33.768793 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-160.ec2.internal\" not found" Jul 29 20:47:33.814613 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.814577 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 20:47:33.876964 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.876932 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" Jul 29 20:47:33.890578 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.890542 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 20:47:33.891952 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.891931 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.ec2.internal" Jul 29 20:47:33.902747 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:33.902720 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 20:47:34.055954 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.055869 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 20:47:34.254957 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.254921 2582 apiserver.go:52] "Watching apiserver" Jul 29 20:47:34.265651 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.265588 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 20:47:34.269408 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.269355 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-zwjfw","kube-system/konnectivity-agent-v4k8h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq","openshift-dns/node-resolver-v5f5q","openshift-image-registry/node-ca-ltkns","openshift-multus/multus-additional-cni-plugins-sxgzz","openshift-network-diagnostics/network-check-target-c287f","openshift-network-operator/iptables-alerter-pnnjn","kube-system/global-pull-secret-syncer-fqq45","kube-system/kube-apiserver-proxy-ip-10-0-137-160.ec2.internal","openshift-cluster-node-tuning-operator/tuned-js997","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal","openshift-multus/multus-nvprw","openshift-multus/network-metrics-daemon-km6rd"] Jul 29 20:47:34.271894 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.271862 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:34.272018 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:34.271948 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:34.273473 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.273449 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.275416 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.275096 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.276445 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.276422 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 20:47:34.276445 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.276440 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 20:47:34.276612 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.276536 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 20:47:34.276679 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.276627 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v5f5q" Jul 29 20:47:34.276734 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.276695 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 20:47:34.276791 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.276758 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-trjrn\"" Jul 29 20:47:34.276984 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.276967 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 20:47:34.277208 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.277191 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 20:47:34.277408 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.277389 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 20:47:34.278227 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.278192 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ltkns" Jul 29 20:47:34.280239 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.280220 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.281924 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.281903 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 20:47:34.282197 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.282178 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tdm7v\"" Jul 29 20:47:34.282557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.282536 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 20:47:34.282824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.282797 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 20:47:34.283028 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.283012 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p28r9\"" Jul 29 20:47:34.283486 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.283392 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:34.283486 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:34.283471 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:34.283697 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.283505 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.285620 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.285600 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v4k8h" Jul 29 20:47:34.289720 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-sys-fs\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.289821 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289731 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:34.289821 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289735 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-run-multus-certs\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.289821 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289760 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-etc-kubernetes\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.289821 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289786 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72e4f020-94dc-446c-9ff1-20a7b4a2f446-serviceca\") pod \"node-ca-ltkns\" (UID: \"72e4f020-94dc-446c-9ff1-20a7b4a2f446\") " pod="openshift-image-registry/node-ca-ltkns" Jul 29 20:47:34.289821 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289815 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-run-netns\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.289821 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:34.289814 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289839 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb6pl\" (UniqueName: \"kubernetes.io/projected/188df71f-e44d-48a7-ab65-f3334dc57092-kube-api-access-lb6pl\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289863 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-device-dir\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72e4f020-94dc-446c-9ff1-20a7b4a2f446-host\") pod \"node-ca-ltkns\" (UID: \"72e4f020-94dc-446c-9ff1-20a7b4a2f446\") " pod="openshift-image-registry/node-ca-ltkns" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289910 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvk2p\" (UniqueName: \"kubernetes.io/projected/72e4f020-94dc-446c-9ff1-20a7b4a2f446-kube-api-access-dvk2p\") pod \"node-ca-ltkns\" (UID: \"72e4f020-94dc-446c-9ff1-20a7b4a2f446\") " pod="openshift-image-registry/node-ca-ltkns" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289935 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-var-lib-cni-bin\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289960 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6hp\" (UniqueName: \"kubernetes.io/projected/4987df3e-bae4-4828-a79f-2cfac383e2d5-kube-api-access-xk6hp\") pod \"node-resolver-v5f5q\" (UID: \"4987df3e-bae4-4828-a79f-2cfac383e2d5\") " pod="openshift-dns/node-resolver-v5f5q" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.289983 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-socket-dir\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290005 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-registration-dir\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290028 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-cnibin\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290052 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-hostroot\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290076 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/188df71f-e44d-48a7-ab65-f3334dc57092-multus-daemon-config\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290125 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.290168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-multus-socket-dir-parent\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290179 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-etc-selinux\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290206 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvhpx\" (UniqueName: \"kubernetes.io/projected/d085f9bf-b95d-487e-953b-b20b22f4226d-kube-api-access-vvhpx\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290233 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-multus-cni-dir\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290268 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-run-k8s-cni-cncf-io\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290318 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-var-lib-kubelet\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290344 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-multus-conf-dir\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290370 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r72m\" (UniqueName: \"kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m\") pod \"network-check-target-c287f\" (UID: \"cfd78950-23f5-4913-a500-8e477de213bd\") " pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/188df71f-e44d-48a7-ab65-f3334dc57092-cni-binary-copy\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290417 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290421 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-var-lib-cni-multus\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290702 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zvfsr\"" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290733 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4987df3e-bae4-4828-a79f-2cfac383e2d5-tmp-dir\") pod \"node-resolver-v5f5q\" (UID: \"4987df3e-bae4-4828-a79f-2cfac383e2d5\") " pod="openshift-dns/node-resolver-v5f5q" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290764 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4987df3e-bae4-4828-a79f-2cfac383e2d5-hosts-file\") pod \"node-resolver-v5f5q\" (UID: \"4987df3e-bae4-4828-a79f-2cfac383e2d5\") " pod="openshift-dns/node-resolver-v5f5q" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290791 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-system-cni-dir\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.290824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290817 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-os-release\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.291675 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.290902 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 20:47:34.291675 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.291158 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 20:47:34.291675 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.291525 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 20:47:34.291675 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.291639 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2ljhc\"" Jul 29 20:47:34.291884 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.291685 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 20:47:34.291884 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.291834 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 20:47:34.292072 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.292053 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pnnjn" Jul 29 20:47:34.293998 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.293978 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 20:47:34.294125 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.294091 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 20:47:34.294197 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.294164 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 20:47:34.294292 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.294272 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 20:47:34.294348 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.294289 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 20:47:34.294403 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.294353 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 20:47:34.294893 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.294863 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 20:47:34.294893 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.294882 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-76l8q\"" Jul 29 20:47:34.295149 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.294925 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 20:47:34.295231 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.295203 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 20:47:34.295309 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.295238 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kq7lr\"" Jul 29 20:47:34.295374 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.295334 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 20:47:34.295503 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.295485 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pq97p\"" Jul 29 20:47:34.295588 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.295572 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hbw52\"" Jul 29 20:47:34.295728 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.295707 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 20:47:34.295941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.295822 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 20:47:34.295941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.295916 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 20:47:34.298999 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.298974 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 20:47:34.378101 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.378066 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 20:47:34.391142 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391113 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-device-dir\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.391291 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72e4f020-94dc-446c-9ff1-20a7b4a2f446-host\") pod \"node-ca-ltkns\" (UID: \"72e4f020-94dc-446c-9ff1-20a7b4a2f446\") " pod="openshift-image-registry/node-ca-ltkns" Jul 29 20:47:34.391291 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391176 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r929\" (UniqueName: \"kubernetes.io/projected/44c3faed-67bf-41d8-b138-dd37c61977f6-kube-api-access-6r929\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.391291 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391207 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-var-lib-kubelet\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.391291 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391233 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6hp\" (UniqueName: \"kubernetes.io/projected/4987df3e-bae4-4828-a79f-2cfac383e2d5-kube-api-access-xk6hp\") pod \"node-resolver-v5f5q\" (UID: \"4987df3e-bae4-4828-a79f-2cfac383e2d5\") " pod="openshift-dns/node-resolver-v5f5q" Jul 29 20:47:34.391291 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391259 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/188df71f-e44d-48a7-ab65-f3334dc57092-multus-daemon-config\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.391291 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391282 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-tuned\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391298 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx5jx\" (UniqueName: \"kubernetes.io/projected/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-kube-api-access-tx5jx\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391315 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-ovn-node-metrics-cert\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-multus-socket-dir-parent\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391348 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44c3faed-67bf-41d8-b138-dd37c61977f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391362 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-host\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391380 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-var-lib-openvswitch\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391400 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-run-openvswitch\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391419 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvhpx\" (UniqueName: \"kubernetes.io/projected/d085f9bf-b95d-487e-953b-b20b22f4226d-kube-api-access-vvhpx\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391458 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-run-k8s-cni-cncf-io\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391485 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-multus-conf-dir\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391511 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44c3faed-67bf-41d8-b138-dd37c61977f6-system-cni-dir\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-kubernetes\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391545 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-slash\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.391557 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-run-systemd\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391574 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-etc-openvswitch\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391588 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44c3faed-67bf-41d8-b138-dd37c61977f6-cnibin\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-dbus\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-tmp\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391650 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-log-socket\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391700 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391727 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6djm\" (UniqueName: \"kubernetes.io/projected/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-kube-api-access-n6djm\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391753 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4987df3e-bae4-4828-a79f-2cfac383e2d5-tmp-dir\") pod \"node-resolver-v5f5q\" (UID: \"4987df3e-bae4-4828-a79f-2cfac383e2d5\") " pod="openshift-dns/node-resolver-v5f5q" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391773 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-system-cni-dir\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-etc-kubernetes\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391833 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/44c3faed-67bf-41d8-b138-dd37c61977f6-os-release\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391857 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391878 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-node-log\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391904 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-ovnkube-config\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-ovnkube-script-lib\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391956 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb523307-f7ba-4f4f-8857-a04d1d7f0938-agent-certs\") pod \"konnectivity-agent-v4k8h\" (UID: \"fb523307-f7ba-4f4f-8857-a04d1d7f0938\") " pod="kube-system/konnectivity-agent-v4k8h" Jul 29 20:47:34.392030 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.391981 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-run-netns\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392005 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44c3faed-67bf-41d8-b138-dd37c61977f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392027 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-sysctl-conf\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392054 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvk2p\" (UniqueName: \"kubernetes.io/projected/72e4f020-94dc-446c-9ff1-20a7b4a2f446-kube-api-access-dvk2p\") pod \"node-ca-ltkns\" (UID: \"72e4f020-94dc-446c-9ff1-20a7b4a2f446\") " pod="openshift-image-registry/node-ca-ltkns" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-var-lib-cni-bin\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44c3faed-67bf-41d8-b138-dd37c61977f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392158 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-socket-dir\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-registration-dir\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392200 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-cnibin\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-hostroot\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392253 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-sysctl-d\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392280 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-systemd-units\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392306 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-run-netns\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392337 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392361 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-sysconfig\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392388 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-run-ovn-kubernetes\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.392819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-cni-netd\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392444 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-etc-selinux\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392504 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-multus-cni-dir\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392531 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-var-lib-kubelet\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392557 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-lib-modules\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392585 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxvc\" (UniqueName: \"kubernetes.io/projected/fe912e79-0992-4a71-a1b7-6237b905a9b9-kube-api-access-qgxvc\") pod \"iptables-alerter-pnnjn\" (UID: \"fe912e79-0992-4a71-a1b7-6237b905a9b9\") " pod="openshift-network-operator/iptables-alerter-pnnjn" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-run-ovn\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392626 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-cni-bin\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r72m\" (UniqueName: \"kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m\") pod \"network-check-target-c287f\" (UID: \"cfd78950-23f5-4913-a500-8e477de213bd\") " pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/188df71f-e44d-48a7-ab65-f3334dc57092-cni-binary-copy\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392680 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-var-lib-cni-multus\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392699 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-kubelet-config\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-modprobe-d\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392732 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb523307-f7ba-4f4f-8857-a04d1d7f0938-konnectivity-ca\") pod \"konnectivity-agent-v4k8h\" (UID: \"fb523307-f7ba-4f4f-8857-a04d1d7f0938\") " pod="kube-system/konnectivity-agent-v4k8h" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392758 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4987df3e-bae4-4828-a79f-2cfac383e2d5-hosts-file\") pod \"node-resolver-v5f5q\" (UID: \"4987df3e-bae4-4828-a79f-2cfac383e2d5\") " pod="openshift-dns/node-resolver-v5f5q" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392774 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-os-release\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.393484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392793 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44c3faed-67bf-41d8-b138-dd37c61977f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392815 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe912e79-0992-4a71-a1b7-6237b905a9b9-iptables-alerter-script\") pod \"iptables-alerter-pnnjn\" (UID: \"fe912e79-0992-4a71-a1b7-6237b905a9b9\") " pod="openshift-network-operator/iptables-alerter-pnnjn" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392836 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-kubelet\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392859 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2drbn\" (UniqueName: \"kubernetes.io/projected/52b760b0-11c0-4512-aa57-fd441bdfd5a1-kube-api-access-2drbn\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-sys-fs\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-run-multus-certs\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392928 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-sys\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392960 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72e4f020-94dc-446c-9ff1-20a7b4a2f446-serviceca\") pod \"node-ca-ltkns\" (UID: \"72e4f020-94dc-446c-9ff1-20a7b4a2f446\") " pod="openshift-image-registry/node-ca-ltkns" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb6pl\" (UniqueName: \"kubernetes.io/projected/188df71f-e44d-48a7-ab65-f3334dc57092-kube-api-access-lb6pl\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.392992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-systemd\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.393008 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-run\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.393024 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe912e79-0992-4a71-a1b7-6237b905a9b9-host-slash\") pod \"iptables-alerter-pnnjn\" (UID: \"fe912e79-0992-4a71-a1b7-6237b905a9b9\") " pod="openshift-network-operator/iptables-alerter-pnnjn" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.393041 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-env-overrides\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.393175 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-device-dir\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.393227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72e4f020-94dc-446c-9ff1-20a7b4a2f446-host\") pod \"node-ca-ltkns\" (UID: \"72e4f020-94dc-446c-9ff1-20a7b4a2f446\") " pod="openshift-image-registry/node-ca-ltkns" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.394185 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/188df71f-e44d-48a7-ab65-f3334dc57092-multus-daemon-config\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.394335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.394325 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-multus-socket-dir-parent\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.395151 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.394615 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-run-k8s-cni-cncf-io\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.395151 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.394649 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-multus-conf-dir\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.395151 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.394995 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4987df3e-bae4-4828-a79f-2cfac383e2d5-tmp-dir\") pod \"node-resolver-v5f5q\" (UID: \"4987df3e-bae4-4828-a79f-2cfac383e2d5\") " pod="openshift-dns/node-resolver-v5f5q" Jul 29 20:47:34.395151 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.395061 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-system-cni-dir\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.395151 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.395128 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-etc-kubernetes\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.395399 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.395222 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-run-netns\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.395453 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.395442 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-var-lib-cni-bin\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.395762 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.395562 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-socket-dir\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.395762 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.395684 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-etc-selinux\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.395762 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.395690 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-run-multus-certs\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.395762 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.395690 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-registration-dir\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.395762 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.395698 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-multus-cni-dir\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.395762 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.395735 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-cnibin\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.395762 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.395761 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-sys-fs\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.396249 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.395895 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4987df3e-bae4-4828-a79f-2cfac383e2d5-hosts-file\") pod \"node-resolver-v5f5q\" (UID: \"4987df3e-bae4-4828-a79f-2cfac383e2d5\") " pod="openshift-dns/node-resolver-v5f5q" Jul 29 20:47:34.396249 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.396049 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-var-lib-kubelet\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.396249 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.396047 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72e4f020-94dc-446c-9ff1-20a7b4a2f446-serviceca\") pod \"node-ca-ltkns\" (UID: \"72e4f020-94dc-446c-9ff1-20a7b4a2f446\") " pod="openshift-image-registry/node-ca-ltkns" Jul 29 20:47:34.396249 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.396114 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-host-var-lib-cni-multus\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.396249 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.396170 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-os-release\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.396249 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.396196 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d085f9bf-b95d-487e-953b-b20b22f4226d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.396249 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.396246 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/188df71f-e44d-48a7-ab65-f3334dc57092-hostroot\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.396487 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.396366 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/188df71f-e44d-48a7-ab65-f3334dc57092-cni-binary-copy\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.401646 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.401615 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sqcr7" Jul 29 20:47:34.403644 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.403613 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 20:47:34.405813 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:34.405499 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:47:34.405813 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:34.405529 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:47:34.405813 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:34.405545 2582 projected.go:196] Error preparing data for projected volume kube-api-access-2r72m for pod openshift-network-diagnostics/network-check-target-c287f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:47:34.405813 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:34.405616 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m podName:cfd78950-23f5-4913-a500-8e477de213bd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:34.905593508 +0000 UTC m=+3.029183344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2r72m" (UniqueName: "kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m") pod "network-check-target-c287f" (UID: "cfd78950-23f5-4913-a500-8e477de213bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:47:34.407595 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.407574 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6hp\" (UniqueName: \"kubernetes.io/projected/4987df3e-bae4-4828-a79f-2cfac383e2d5-kube-api-access-xk6hp\") pod \"node-resolver-v5f5q\" (UID: \"4987df3e-bae4-4828-a79f-2cfac383e2d5\") " pod="openshift-dns/node-resolver-v5f5q" Jul 29 20:47:34.408229 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.408205 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sqcr7" Jul 29 20:47:34.409015 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.408534 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvhpx\" (UniqueName: \"kubernetes.io/projected/d085f9bf-b95d-487e-953b-b20b22f4226d-kube-api-access-vvhpx\") pod \"aws-ebs-csi-driver-node-8x8qq\" (UID: \"d085f9bf-b95d-487e-953b-b20b22f4226d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.409015 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.408971 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvk2p\" (UniqueName: \"kubernetes.io/projected/72e4f020-94dc-446c-9ff1-20a7b4a2f446-kube-api-access-dvk2p\") pod \"node-ca-ltkns\" (UID: \"72e4f020-94dc-446c-9ff1-20a7b4a2f446\") " pod="openshift-image-registry/node-ca-ltkns" Jul 29 20:47:34.409495 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.409474 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb6pl\" (UniqueName: \"kubernetes.io/projected/188df71f-e44d-48a7-ab65-f3334dc57092-kube-api-access-lb6pl\") pod \"multus-nvprw\" (UID: \"188df71f-e44d-48a7-ab65-f3334dc57092\") " pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.494048 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494006 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6djm\" (UniqueName: \"kubernetes.io/projected/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-kube-api-access-n6djm\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.494233 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494080 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/44c3faed-67bf-41d8-b138-dd37c61977f6-os-release\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.494233 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494110 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:34.494233 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:34.494192 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:47:34.494337 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:34.494252 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret podName:2c45ab5d-799b-4ffc-b8e1-40fc891d4d87 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:34.994237649 +0000 UTC m=+3.117827471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret") pod "global-pull-secret-syncer-fqq45" (UID: "2c45ab5d-799b-4ffc-b8e1-40fc891d4d87") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:47:34.494400 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494381 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/44c3faed-67bf-41d8-b138-dd37c61977f6-os-release\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.494479 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-node-log\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.494479 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494434 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-node-log\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.494479 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494444 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-ovnkube-config\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.494479 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494468 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-ovnkube-script-lib\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.494653 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494517 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb523307-f7ba-4f4f-8857-a04d1d7f0938-agent-certs\") pod \"konnectivity-agent-v4k8h\" (UID: \"fb523307-f7ba-4f4f-8857-a04d1d7f0938\") " pod="kube-system/konnectivity-agent-v4k8h" Jul 29 20:47:34.494653 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44c3faed-67bf-41d8-b138-dd37c61977f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.494653 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494577 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-sysctl-conf\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.494653 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494604 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44c3faed-67bf-41d8-b138-dd37c61977f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.494653 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494634 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-sysctl-d\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.494854 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-systemd-units\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.494854 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-run-netns\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.494854 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-sysconfig\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.494854 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494733 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-run-ovn-kubernetes\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.494854 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-cni-netd\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.494854 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494788 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:34.494854 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494817 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-lib-modules\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.494854 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494843 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxvc\" (UniqueName: \"kubernetes.io/projected/fe912e79-0992-4a71-a1b7-6237b905a9b9-kube-api-access-qgxvc\") pod \"iptables-alerter-pnnjn\" (UID: \"fe912e79-0992-4a71-a1b7-6237b905a9b9\") " pod="openshift-network-operator/iptables-alerter-pnnjn" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494869 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-run-ovn\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494894 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-cni-bin\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494935 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-kubelet-config\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494961 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-modprobe-d\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.494987 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb523307-f7ba-4f4f-8857-a04d1d7f0938-konnectivity-ca\") pod \"konnectivity-agent-v4k8h\" (UID: \"fb523307-f7ba-4f4f-8857-a04d1d7f0938\") " pod="kube-system/konnectivity-agent-v4k8h" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44c3faed-67bf-41d8-b138-dd37c61977f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495042 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe912e79-0992-4a71-a1b7-6237b905a9b9-iptables-alerter-script\") pod \"iptables-alerter-pnnjn\" (UID: \"fe912e79-0992-4a71-a1b7-6237b905a9b9\") " pod="openshift-network-operator/iptables-alerter-pnnjn" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495066 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-kubelet\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495092 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2drbn\" (UniqueName: \"kubernetes.io/projected/52b760b0-11c0-4512-aa57-fd441bdfd5a1-kube-api-access-2drbn\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495119 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-ovnkube-config\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-sys\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495169 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-systemd\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495192 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-run\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:34.495208 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495231 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe912e79-0992-4a71-a1b7-6237b905a9b9-host-slash\") pod \"iptables-alerter-pnnjn\" (UID: \"fe912e79-0992-4a71-a1b7-6237b905a9b9\") " pod="openshift-network-operator/iptables-alerter-pnnjn" Jul 29 20:47:34.495253 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:34.495258 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs podName:52b760b0-11c0-4512-aa57-fd441bdfd5a1 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:34.995242261 +0000 UTC m=+3.118832088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs") pod "network-metrics-daemon-km6rd" (UID: "52b760b0-11c0-4512-aa57-fd441bdfd5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-env-overrides\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495292 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe912e79-0992-4a71-a1b7-6237b905a9b9-host-slash\") pod \"iptables-alerter-pnnjn\" (UID: \"fe912e79-0992-4a71-a1b7-6237b905a9b9\") " pod="openshift-network-operator/iptables-alerter-pnnjn" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495317 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r929\" (UniqueName: \"kubernetes.io/projected/44c3faed-67bf-41d8-b138-dd37c61977f6-kube-api-access-6r929\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495346 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-var-lib-kubelet\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-tuned\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495401 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx5jx\" (UniqueName: \"kubernetes.io/projected/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-kube-api-access-tx5jx\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495428 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-ovn-node-metrics-cert\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495450 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-lib-modules\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495455 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44c3faed-67bf-41d8-b138-dd37c61977f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495480 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-host\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495535 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-host\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495684 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-run-ovn\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495739 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-cni-bin\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495797 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-kubelet-config\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:34.495941 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.495893 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-modprobe-d\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.496527 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.496009 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-ovnkube-script-lib\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.496527 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.496355 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb523307-f7ba-4f4f-8857-a04d1d7f0938-konnectivity-ca\") pod \"konnectivity-agent-v4k8h\" (UID: \"fb523307-f7ba-4f4f-8857-a04d1d7f0938\") " pod="kube-system/konnectivity-agent-v4k8h" Jul 29 20:47:34.496527 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.496368 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-cni-netd\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.496655 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.496566 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44c3faed-67bf-41d8-b138-dd37c61977f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.497011 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.496972 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe912e79-0992-4a71-a1b7-6237b905a9b9-iptables-alerter-script\") pod \"iptables-alerter-pnnjn\" (UID: \"fe912e79-0992-4a71-a1b7-6237b905a9b9\") " pod="openshift-network-operator/iptables-alerter-pnnjn" Jul 29 20:47:34.497143 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497075 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-kubelet\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.497143 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497121 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44c3faed-67bf-41d8-b138-dd37c61977f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.497262 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497244 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-env-overrides\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.497321 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497271 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-sysctl-conf\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.497321 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497301 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-systemd\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.497321 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497313 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-sys\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.497470 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497340 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-run\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.497470 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-var-lib-openvswitch\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.497470 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-run-openvswitch\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.497470 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44c3faed-67bf-41d8-b138-dd37c61977f6-system-cni-dir\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.497470 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497422 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-kubernetes\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.497470 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-slash\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.497470 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-run-systemd\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497478 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-etc-openvswitch\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497495 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44c3faed-67bf-41d8-b138-dd37c61977f6-cnibin\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497511 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-dbus\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-tmp\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497545 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-log-socket\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497566 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497568 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-var-lib-kubelet\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497617 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-var-lib-openvswitch\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497656 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-run-openvswitch\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497681 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44c3faed-67bf-41d8-b138-dd37c61977f6-system-cni-dir\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497696 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44c3faed-67bf-41d8-b138-dd37c61977f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-kubernetes\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497743 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-slash\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.497770 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497766 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-run-systemd\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.498365 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497785 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-sysctl-d\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.498365 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497788 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-etc-openvswitch\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.498365 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497814 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-systemd-units\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.498365 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497822 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44c3faed-67bf-41d8-b138-dd37c61977f6-cnibin\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.498365 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497846 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-run-netns\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.498365 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497892 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-sysconfig\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.498365 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497919 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-dbus\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:34.498365 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497929 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-run-ovn-kubernetes\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.498365 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.497974 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.498365 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.498091 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-log-socket\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.498855 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.498547 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44c3faed-67bf-41d8-b138-dd37c61977f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.500842 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.500809 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb523307-f7ba-4f4f-8857-a04d1d7f0938-agent-certs\") pod \"konnectivity-agent-v4k8h\" (UID: \"fb523307-f7ba-4f4f-8857-a04d1d7f0938\") " pod="kube-system/konnectivity-agent-v4k8h" Jul 29 20:47:34.501543 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.501503 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-ovn-node-metrics-cert\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.507681 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.507630 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6djm\" (UniqueName: \"kubernetes.io/projected/e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e-kube-api-access-n6djm\") pod \"ovnkube-node-zwjfw\" (UID: \"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.509078 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.509036 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx5jx\" (UniqueName: \"kubernetes.io/projected/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-kube-api-access-tx5jx\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.509340 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.509288 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2drbn\" (UniqueName: \"kubernetes.io/projected/52b760b0-11c0-4512-aa57-fd441bdfd5a1-kube-api-access-2drbn\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:34.509904 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.509883 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-etc-tuned\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.511065 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.511035 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r929\" (UniqueName: \"kubernetes.io/projected/44c3faed-67bf-41d8-b138-dd37c61977f6-kube-api-access-6r929\") pod \"multus-additional-cni-plugins-sxgzz\" (UID: \"44c3faed-67bf-41d8-b138-dd37c61977f6\") " pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.511786 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.511566 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxvc\" (UniqueName: \"kubernetes.io/projected/fe912e79-0992-4a71-a1b7-6237b905a9b9-kube-api-access-qgxvc\") pod \"iptables-alerter-pnnjn\" (UID: \"fe912e79-0992-4a71-a1b7-6237b905a9b9\") " pod="openshift-network-operator/iptables-alerter-pnnjn" Jul 29 20:47:34.511786 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.511706 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6d8dec5-f8a5-4f43-bf15-118cfcf70e42-tmp\") pod \"tuned-js997\" (UID: \"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42\") " pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.592655 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.592571 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nvprw" Jul 29 20:47:34.604974 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.604944 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" Jul 29 20:47:34.617160 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.617136 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v5f5q" Jul 29 20:47:34.625484 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.625464 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ltkns" Jul 29 20:47:34.632808 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.632782 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sxgzz" Jul 29 20:47:34.641325 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.641299 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:34.650657 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.650636 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v4k8h" Jul 29 20:47:34.666888 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.666860 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-js997" Jul 29 20:47:34.676014 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:34.675991 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pnnjn" Jul 29 20:47:35.001889 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:35.001792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:35.001889 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:35.001853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:35.002140 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:35.001889 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r72m\" (UniqueName: \"kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m\") pod \"network-check-target-c287f\" (UID: \"cfd78950-23f5-4913-a500-8e477de213bd\") " pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:35.002140 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:35.001967 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:47:35.002140 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:35.002030 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:47:35.002140 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:35.002046 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:47:35.002140 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:35.001972 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:47:35.002140 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:35.002061 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret podName:2c45ab5d-799b-4ffc-b8e1-40fc891d4d87 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:36.002018855 +0000 UTC m=+4.125608713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret") pod "global-pull-secret-syncer-fqq45" (UID: "2c45ab5d-799b-4ffc-b8e1-40fc891d4d87") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:47:35.002140 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:35.002077 2582 projected.go:196] Error preparing data for projected volume kube-api-access-2r72m for pod openshift-network-diagnostics/network-check-target-c287f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:47:35.002140 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:35.002101 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs podName:52b760b0-11c0-4512-aa57-fd441bdfd5a1 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:36.002086347 +0000 UTC m=+4.125676170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs") pod "network-metrics-daemon-km6rd" (UID: "52b760b0-11c0-4512-aa57-fd441bdfd5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:47:35.002140 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:35.002123 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m podName:cfd78950-23f5-4913-a500-8e477de213bd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:36.002108062 +0000 UTC m=+4.125697886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2r72m" (UniqueName: "kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m") pod "network-check-target-c287f" (UID: "cfd78950-23f5-4913-a500-8e477de213bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:47:35.412053 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:35.411940 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 20:42:34 +0000 UTC" deadline="2028-01-04 14:16:41.467290382 +0000 UTC" Jul 29 20:47:35.412053 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:35.411981 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12569h29m6.055313961s" Jul 29 20:47:35.486818 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:35.486792 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:35.486903 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:35.486828 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:35.486949 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:35.486919 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:35.487034 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:35.487014 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:35.717413 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:35.716982 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ad47a9_1bdd_4d86_b0e1_fd05c2d4190e.slice/crio-63cd109ff77a60dbd8455245700f1483d46a85218a2b14389d97c8b7fae1c0d6 WatchSource:0}: Error finding container 63cd109ff77a60dbd8455245700f1483d46a85218a2b14389d97c8b7fae1c0d6: Status 404 returned error can't find the container with id 63cd109ff77a60dbd8455245700f1483d46a85218a2b14389d97c8b7fae1c0d6 Jul 29 20:47:35.718853 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:35.718823 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd085f9bf_b95d_487e_953b_b20b22f4226d.slice/crio-0306768f33b8c4cc9ab926a919a16ee4be7036e89f4cd5d753a498821b346cd9 WatchSource:0}: Error finding container 0306768f33b8c4cc9ab926a919a16ee4be7036e89f4cd5d753a498821b346cd9: Status 404 returned error can't find the container with id 0306768f33b8c4cc9ab926a919a16ee4be7036e89f4cd5d753a498821b346cd9 Jul 29 20:47:35.719374 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:35.719345 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe912e79_0992_4a71_a1b7_6237b905a9b9.slice/crio-d5a1096ea887cb6ac450bc92348100f2c809134aec2af224203bf459592d6cc3 WatchSource:0}: Error finding container d5a1096ea887cb6ac450bc92348100f2c809134aec2af224203bf459592d6cc3: Status 404 returned error can't find the container with id d5a1096ea887cb6ac450bc92348100f2c809134aec2af224203bf459592d6cc3 Jul 29 20:47:35.722824 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:35.722807 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c3faed_67bf_41d8_b138_dd37c61977f6.slice/crio-b9e0fa416a595a2c513e500b571b8be2b5a25aa9890e145b9fbaab4c5a52332b WatchSource:0}: Error finding container b9e0fa416a595a2c513e500b571b8be2b5a25aa9890e145b9fbaab4c5a52332b: Status 404 returned error can't find the container with id b9e0fa416a595a2c513e500b571b8be2b5a25aa9890e145b9fbaab4c5a52332b Jul 29 20:47:35.724466 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:35.724447 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d8dec5_f8a5_4f43_bf15_118cfcf70e42.slice/crio-d506670a80f7f6cf69c2c0eed8cc09efa9414050f7da4da474624ab03ba6a17a WatchSource:0}: Error finding container d506670a80f7f6cf69c2c0eed8cc09efa9414050f7da4da474624ab03ba6a17a: Status 404 returned error can't find the container with id d506670a80f7f6cf69c2c0eed8cc09efa9414050f7da4da474624ab03ba6a17a Jul 29 20:47:35.726078 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:35.725941 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb523307_f7ba_4f4f_8857_a04d1d7f0938.slice/crio-cb81a63c81352a83ae7efadca7d1b4383788a96518b0fd79c444a75aeb937cac WatchSource:0}: Error finding container cb81a63c81352a83ae7efadca7d1b4383788a96518b0fd79c444a75aeb937cac: Status 404 returned error can't find the container with id cb81a63c81352a83ae7efadca7d1b4383788a96518b0fd79c444a75aeb937cac Jul 29 20:47:35.726808 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:35.726672 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e4f020_94dc_446c_9ff1_20a7b4a2f446.slice/crio-002a64d8826490f641272e47dc2e53f592d621fa6f96eb096551db1f730e0f90 WatchSource:0}: Error finding container 002a64d8826490f641272e47dc2e53f592d621fa6f96eb096551db1f730e0f90: Status 404 returned error can't find the container with id 002a64d8826490f641272e47dc2e53f592d621fa6f96eb096551db1f730e0f90 Jul 29 20:47:35.728229 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:35.728200 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188df71f_e44d_48a7_ab65_f3334dc57092.slice/crio-76f1914b5f18352195997f8aed12fe7d5d64077bff00d7d3c25679886f65e9a3 WatchSource:0}: Error finding container 76f1914b5f18352195997f8aed12fe7d5d64077bff00d7d3c25679886f65e9a3: Status 404 returned error can't find the container with id 76f1914b5f18352195997f8aed12fe7d5d64077bff00d7d3c25679886f65e9a3 Jul 29 20:47:35.729545 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:47:35.729517 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4987df3e_bae4_4828_a79f_2cfac383e2d5.slice/crio-edbd99d21d034b3323b35eee83b7768508acf3801eb1b3d0baf978b5f4e8b027 WatchSource:0}: Error finding container edbd99d21d034b3323b35eee83b7768508acf3801eb1b3d0baf978b5f4e8b027: Status 404 returned error can't find the container with id edbd99d21d034b3323b35eee83b7768508acf3801eb1b3d0baf978b5f4e8b027 Jul 29 20:47:36.010433 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.010400 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:36.010556 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.010443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r72m\" (UniqueName: \"kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m\") pod \"network-check-target-c287f\" (UID: \"cfd78950-23f5-4913-a500-8e477de213bd\") " pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:36.010556 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.010489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:36.010659 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:36.010576 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:47:36.010659 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:36.010634 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret podName:2c45ab5d-799b-4ffc-b8e1-40fc891d4d87 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:38.010620371 +0000 UTC m=+6.134210191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret") pod "global-pull-secret-syncer-fqq45" (UID: "2c45ab5d-799b-4ffc-b8e1-40fc891d4d87") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:47:36.010930 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:36.010916 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:47:36.010965 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:36.010955 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs podName:52b760b0-11c0-4512-aa57-fd441bdfd5a1 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:38.010944557 +0000 UTC m=+6.134534379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs") pod "network-metrics-daemon-km6rd" (UID: "52b760b0-11c0-4512-aa57-fd441bdfd5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:47:36.011022 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:36.011012 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:47:36.011068 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:36.011028 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:47:36.011068 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:36.011038 2582 projected.go:196] Error preparing data for projected volume kube-api-access-2r72m for pod openshift-network-diagnostics/network-check-target-c287f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:47:36.011068 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:36.011063 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m podName:cfd78950-23f5-4913-a500-8e477de213bd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:38.011056144 +0000 UTC m=+6.134645965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2r72m" (UniqueName: "kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m") pod "network-check-target-c287f" (UID: "cfd78950-23f5-4913-a500-8e477de213bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:47:36.412696 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.412545 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 20:42:34 +0000 UTC" deadline="2028-02-29 00:21:09.575062079 +0000 UTC" Jul 29 20:47:36.412696 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.412610 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13899h33m33.162456208s" Jul 29 20:47:36.489972 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.489937 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:36.490189 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:36.490093 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:36.511028 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.510987 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.ec2.internal" event={"ID":"8e6a9ef8e0380c11ab0ad572fa464511","Type":"ContainerStarted","Data":"c05398a3d64babc1439b19a989f6b2759c73f007e8786671dbbeda9875414031"} Jul 29 20:47:36.518379 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.518342 2582 generic.go:356] "Generic (PLEG): container finished" podID="21c1930d16639ed349b597b7c5397dfb" containerID="e5c69381fbaae0d174ec6d5f17abbf398be3b20234c465e2c5ad6d3ec2841606" exitCode=0 Jul 29 20:47:36.518544 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.518432 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" event={"ID":"21c1930d16639ed349b597b7c5397dfb","Type":"ContainerDied","Data":"e5c69381fbaae0d174ec6d5f17abbf398be3b20234c465e2c5ad6d3ec2841606"} Jul 29 20:47:36.525690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.525657 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v5f5q" event={"ID":"4987df3e-bae4-4828-a79f-2cfac383e2d5","Type":"ContainerStarted","Data":"edbd99d21d034b3323b35eee83b7768508acf3801eb1b3d0baf978b5f4e8b027"} Jul 29 20:47:36.530526 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.530489 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ltkns" event={"ID":"72e4f020-94dc-446c-9ff1-20a7b4a2f446","Type":"ContainerStarted","Data":"002a64d8826490f641272e47dc2e53f592d621fa6f96eb096551db1f730e0f90"} Jul 29 20:47:36.541731 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.541674 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.ec2.internal" podStartSLOduration=2.541653148 podStartE2EDuration="2.541653148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:36.527124631 +0000 UTC m=+4.650714477" watchObservedRunningTime="2026-07-29 20:47:36.541653148 +0000 UTC m=+4.665242999" Jul 29 20:47:36.542870 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.542822 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nvprw" event={"ID":"188df71f-e44d-48a7-ab65-f3334dc57092","Type":"ContainerStarted","Data":"76f1914b5f18352195997f8aed12fe7d5d64077bff00d7d3c25679886f65e9a3"} Jul 29 20:47:36.556353 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.556323 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v4k8h" event={"ID":"fb523307-f7ba-4f4f-8857-a04d1d7f0938","Type":"ContainerStarted","Data":"cb81a63c81352a83ae7efadca7d1b4383788a96518b0fd79c444a75aeb937cac"} Jul 29 20:47:36.563732 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.563697 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-js997" event={"ID":"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42","Type":"ContainerStarted","Data":"d506670a80f7f6cf69c2c0eed8cc09efa9414050f7da4da474624ab03ba6a17a"} Jul 29 20:47:36.572287 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.570426 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxgzz" event={"ID":"44c3faed-67bf-41d8-b138-dd37c61977f6","Type":"ContainerStarted","Data":"b9e0fa416a595a2c513e500b571b8be2b5a25aa9890e145b9fbaab4c5a52332b"} Jul 29 20:47:36.576470 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.576210 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pnnjn" event={"ID":"fe912e79-0992-4a71-a1b7-6237b905a9b9","Type":"ContainerStarted","Data":"d5a1096ea887cb6ac450bc92348100f2c809134aec2af224203bf459592d6cc3"} Jul 29 20:47:36.594910 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.594792 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" event={"ID":"d085f9bf-b95d-487e-953b-b20b22f4226d","Type":"ContainerStarted","Data":"0306768f33b8c4cc9ab926a919a16ee4be7036e89f4cd5d753a498821b346cd9"} Jul 29 20:47:36.600295 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:36.600257 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" event={"ID":"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e","Type":"ContainerStarted","Data":"63cd109ff77a60dbd8455245700f1483d46a85218a2b14389d97c8b7fae1c0d6"} Jul 29 20:47:37.488515 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:37.488475 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:37.489037 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:37.488632 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:37.489144 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:37.489045 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:37.489217 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:37.489169 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:37.618222 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:37.618176 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" event={"ID":"21c1930d16639ed349b597b7c5397dfb","Type":"ContainerStarted","Data":"f45b32c0b15ca93ca830efe4d742d95d71ac939f6d337531bf1299dac6523d95"} Jul 29 20:47:38.028834 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:38.028793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:38.029026 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:38.028869 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:38.029026 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:38.028940 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r72m\" (UniqueName: \"kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m\") pod \"network-check-target-c287f\" (UID: \"cfd78950-23f5-4913-a500-8e477de213bd\") " pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:38.029413 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:38.029390 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:47:38.029502 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:38.029472 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret podName:2c45ab5d-799b-4ffc-b8e1-40fc891d4d87 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:42.029451693 +0000 UTC m=+10.153041523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret") pod "global-pull-secret-syncer-fqq45" (UID: "2c45ab5d-799b-4ffc-b8e1-40fc891d4d87") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:47:38.030026 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:38.030003 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:47:38.030129 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:38.030069 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs podName:52b760b0-11c0-4512-aa57-fd441bdfd5a1 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:42.030050053 +0000 UTC m=+10.153639897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs") pod "network-metrics-daemon-km6rd" (UID: "52b760b0-11c0-4512-aa57-fd441bdfd5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:47:38.030283 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:38.030259 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:47:38.030350 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:38.030285 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:47:38.030350 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:38.030300 2582 projected.go:196] Error preparing data for projected volume kube-api-access-2r72m for pod openshift-network-diagnostics/network-check-target-c287f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:47:38.030452 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:38.030353 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m podName:cfd78950-23f5-4913-a500-8e477de213bd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:42.030337164 +0000 UTC m=+10.153927001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2r72m" (UniqueName: "kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m") pod "network-check-target-c287f" (UID: "cfd78950-23f5-4913-a500-8e477de213bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:47:38.487606 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:38.487525 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:38.487768 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:38.487685 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:39.487039 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:39.486976 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:39.487504 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:39.487136 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:39.487504 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:39.487223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:39.487504 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:39.487300 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:40.488850 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:40.488313 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:40.488850 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:40.488464 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:41.487641 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:41.487608 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:41.487641 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:41.487633 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:41.487900 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:41.487746 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:41.487900 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:41.487859 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:42.062375 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:42.062345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:42.062815 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:42.062402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:42.062815 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:42.062434 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r72m\" (UniqueName: \"kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m\") pod \"network-check-target-c287f\" (UID: \"cfd78950-23f5-4913-a500-8e477de213bd\") " pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:42.062815 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:42.062585 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:47:42.062815 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:42.062603 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:47:42.062815 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:42.062614 2582 projected.go:196] Error preparing data for projected volume kube-api-access-2r72m for pod openshift-network-diagnostics/network-check-target-c287f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:47:42.062815 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:42.062660 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m podName:cfd78950-23f5-4913-a500-8e477de213bd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:50.062647372 +0000 UTC m=+18.186237192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2r72m" (UniqueName: "kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m") pod "network-check-target-c287f" (UID: "cfd78950-23f5-4913-a500-8e477de213bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:47:42.063225 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:42.063121 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:47:42.063225 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:42.063137 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:47:42.063225 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:42.063187 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs podName:52b760b0-11c0-4512-aa57-fd441bdfd5a1 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:50.063169014 +0000 UTC m=+18.186758851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs") pod "network-metrics-daemon-km6rd" (UID: "52b760b0-11c0-4512-aa57-fd441bdfd5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:47:42.063225 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:42.063211 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret podName:2c45ab5d-799b-4ffc-b8e1-40fc891d4d87 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:50.063200929 +0000 UTC m=+18.186790752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret") pod "global-pull-secret-syncer-fqq45" (UID: "2c45ab5d-799b-4ffc-b8e1-40fc891d4d87") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:47:42.488870 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:42.488331 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:42.488870 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:42.488481 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:43.487204 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:43.487136 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:43.487204 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:43.487160 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:43.487719 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:43.487275 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:43.487719 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:43.487369 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:44.488059 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:44.487211 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:44.488059 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:44.487353 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:45.487400 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:45.487368 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:45.487591 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:45.487499 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:45.487591 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:45.487557 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:45.487706 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:45.487671 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:46.486749 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:46.486716 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:46.487236 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:46.486859 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:47.487619 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:47.487586 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:47.488075 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:47.487628 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:47.488075 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:47.487698 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:47.488075 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:47.487839 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:48.486940 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:48.486907 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:48.487142 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:48.487070 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:49.487329 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:49.487293 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:49.487800 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:49.487299 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:49.487800 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:49.487430 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:49.487800 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:49.487494 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:50.124583 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:50.124545 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r72m\" (UniqueName: \"kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m\") pod \"network-check-target-c287f\" (UID: \"cfd78950-23f5-4913-a500-8e477de213bd\") " pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:50.124830 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:50.124621 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:50.124830 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:50.124661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:50.124830 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:50.124732 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:47:50.124830 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:50.124753 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:47:50.124830 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:50.124765 2582 projected.go:196] Error preparing data for projected volume kube-api-access-2r72m for pod openshift-network-diagnostics/network-check-target-c287f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:47:50.124830 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:50.124770 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:47:50.124830 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:50.124777 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:47:50.124830 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:50.124831 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m podName:cfd78950-23f5-4913-a500-8e477de213bd nodeName:}" failed. No retries permitted until 2026-07-29 20:48:06.124812898 +0000 UTC m=+34.248402718 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2r72m" (UniqueName: "kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m") pod "network-check-target-c287f" (UID: "cfd78950-23f5-4913-a500-8e477de213bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:47:50.125162 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:50.124850 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret podName:2c45ab5d-799b-4ffc-b8e1-40fc891d4d87 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:06.124842314 +0000 UTC m=+34.248432135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret") pod "global-pull-secret-syncer-fqq45" (UID: "2c45ab5d-799b-4ffc-b8e1-40fc891d4d87") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:47:50.125162 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:50.124861 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs podName:52b760b0-11c0-4512-aa57-fd441bdfd5a1 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:06.124855167 +0000 UTC m=+34.248444988 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs") pod "network-metrics-daemon-km6rd" (UID: "52b760b0-11c0-4512-aa57-fd441bdfd5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:47:50.487764 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:50.487718 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:50.488176 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:50.487883 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:51.487737 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:51.487688 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:51.487737 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:51.487726 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:51.488299 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:51.487818 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:51.488299 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:51.487951 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:52.488437 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:52.488393 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:52.488883 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:52.488515 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:53.487447 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:53.487410 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:53.487626 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:53.487410 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:53.487626 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:53.487559 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:53.487722 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:53.487639 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:54.487258 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:54.487065 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:54.487642 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:54.487379 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:54.665337 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:54.665202 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-js997" event={"ID":"f6d8dec5-f8a5-4f43-bf15-118cfcf70e42","Type":"ContainerStarted","Data":"ce98115d5be341e9b25019e28301b96418d82452a25e527f11486426315be385"} Jul 29 20:47:54.669837 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:54.669718 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxgzz" event={"ID":"44c3faed-67bf-41d8-b138-dd37c61977f6","Type":"ContainerStarted","Data":"4a5737f7ba0477af925917072a8e06f63856b431d6fa16d05d0a4221917ae675"} Jul 29 20:47:54.671486 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:54.671302 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" event={"ID":"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e","Type":"ContainerStarted","Data":"bbb2af2a706ba912998adcb08c194cd51272deb9afa3be846f488d2d9e92c93a"} Jul 29 20:47:54.677327 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:54.677264 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nvprw" event={"ID":"188df71f-e44d-48a7-ab65-f3334dc57092","Type":"ContainerStarted","Data":"ad88a2eaad212d5027274bedd4332fbb4b13e9327294ae26cc519c6831ecae2e"} Jul 29 20:47:54.679375 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:54.679349 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v4k8h" event={"ID":"fb523307-f7ba-4f4f-8857-a04d1d7f0938","Type":"ContainerStarted","Data":"5863404dc9390a83283f5915cedb0695a445bcff7f53f9fde1a3279a7ba5ed96"} Jul 29 20:47:54.683823 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:54.683770 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.ec2.internal" podStartSLOduration=20.683752427 podStartE2EDuration="20.683752427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:37.63655499 +0000 UTC m=+5.760144837" watchObservedRunningTime="2026-07-29 20:47:54.683752427 +0000 UTC m=+22.807342262" Jul 29 20:47:54.705476 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:54.705430 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-js997" podStartSLOduration=3.962411152 podStartE2EDuration="22.705402935s" totalImagesPullingTime="18.742991783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:54.683909172 +0000 UTC m=+22.807499002" watchObservedRunningTime="2026-07-29 20:47:54.705402935 +0000 UTC m=+22.828992810" Jul 29 20:47:54.705630 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:54.705599 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nvprw" podStartSLOduration=3.9049554349999998 podStartE2EDuration="22.705593141s" totalImagesPullingTime="18.800637706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:54.704847007 +0000 UTC m=+22.828436851" watchObservedRunningTime="2026-07-29 20:47:54.705593141 +0000 UTC m=+22.829182982" Jul 29 20:47:54.719998 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:54.719910 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v4k8h" podStartSLOduration=7.811673024 podStartE2EDuration="22.719889408s" totalImagesPullingTime="14.908216384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:54.719621828 +0000 UTC m=+22.843211670" watchObservedRunningTime="2026-07-29 20:47:54.719889408 +0000 UTC m=+22.843479252" Jul 29 20:47:54.734757 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:54.734540 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v5f5q" podStartSLOduration=4.000707275 podStartE2EDuration="22.734518849s" totalImagesPullingTime="18.733811574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:54.733358883 +0000 UTC m=+22.856948726" watchObservedRunningTime="2026-07-29 20:47:54.734518849 +0000 UTC m=+22.858108692" Jul 29 20:47:54.747694 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:54.747649 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ltkns" podStartSLOduration=4.012711253 podStartE2EDuration="22.747633765s" totalImagesPullingTime="18.734922512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:54.747612832 +0000 UTC m=+22.871202675" watchObservedRunningTime="2026-07-29 20:47:54.747633765 +0000 UTC m=+22.871223605" Jul 29 20:47:55.465187 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.465158 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v4k8h" Jul 29 20:47:55.487440 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.487417 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:55.488169 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:55.487511 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:55.488169 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.487573 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:55.488169 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:55.487658 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:55.682051 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.682002 2582 generic.go:356] "Generic (PLEG): container finished" podID="44c3faed-67bf-41d8-b138-dd37c61977f6" containerID="4a5737f7ba0477af925917072a8e06f63856b431d6fa16d05d0a4221917ae675" exitCode=0 Jul 29 20:47:55.682340 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.682095 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxgzz" event={"ID":"44c3faed-67bf-41d8-b138-dd37c61977f6","Type":"ContainerDied","Data":"4a5737f7ba0477af925917072a8e06f63856b431d6fa16d05d0a4221917ae675"} Jul 29 20:47:55.683640 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.683576 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" event={"ID":"d085f9bf-b95d-487e-953b-b20b22f4226d","Type":"ContainerStarted","Data":"84559878f23027f1808e3e3b0c43c2834c01ea47740c1f8c041286969f9af913"} Jul 29 20:47:55.686511 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.686489 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" event={"ID":"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e","Type":"ContainerStarted","Data":"2ca126c190d8f44930c452964e103c42f0c560e5f53f120a3dc77651524477e7"} Jul 29 20:47:55.686644 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.686566 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" event={"ID":"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e","Type":"ContainerStarted","Data":"d2326cdddf0b3b67455e5ca136c10ef78f0598245a4f1b0023dd541101e8bd73"} Jul 29 20:47:55.686644 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.686630 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" event={"ID":"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e","Type":"ContainerStarted","Data":"ca7095741d3038725c966bbfcc51e3f6b5b218e0730b28463dcf1027210f78c6"} Jul 29 20:47:55.686783 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.686677 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" event={"ID":"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e","Type":"ContainerStarted","Data":"ace908e19f5de98fa5c1101c655feccd4a25b42a98ccb87b81d7a3314a2a432f"} Jul 29 20:47:55.686783 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.686709 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" event={"ID":"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e","Type":"ContainerStarted","Data":"ffc8b5638205bb408b1c30712879f827a91315f152289cdbdd84e2fd2365c748"} Jul 29 20:47:55.687889 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.687870 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v5f5q" event={"ID":"4987df3e-bae4-4828-a79f-2cfac383e2d5","Type":"ContainerStarted","Data":"935e152b6faf9d0fc67c9fb0fbd55b338fbee9b49730eca97f25e36861474482"} Jul 29 20:47:55.689245 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:55.689226 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ltkns" event={"ID":"72e4f020-94dc-446c-9ff1-20a7b4a2f446","Type":"ContainerStarted","Data":"112abce5a4d1dd33a72b955544184101fcd14e4d25b0b9e290c548350e1b9ba1"} Jul 29 20:47:56.047156 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:56.047130 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 20:47:56.351046 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:56.350841 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T20:47:56.047151414Z","UUID":"c48f2a0f-02a6-4ca6-a763-6fc7f5d53bb2","Handler":null,"Name":"","Endpoint":""} Jul 29 20:47:56.352746 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:56.352708 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 20:47:56.352872 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:56.352755 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 20:47:56.487284 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:56.487252 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:56.487469 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:56.487413 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:56.693186 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:56.693102 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pnnjn" event={"ID":"fe912e79-0992-4a71-a1b7-6237b905a9b9","Type":"ContainerStarted","Data":"be3324ec70a3eb82d6778836e686a32d18da72ce2ca803e7d06e0bd983a6efa4"} Jul 29 20:47:56.695832 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:56.695793 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" event={"ID":"d085f9bf-b95d-487e-953b-b20b22f4226d","Type":"ContainerStarted","Data":"576f306082a5751623cf3202d4203252adfc602e57fb2c13a0fb9ebd528003a2"} Jul 29 20:47:56.707920 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:56.707866 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pnnjn" podStartSLOduration=5.962294669 podStartE2EDuration="24.707848004s" totalImagesPullingTime="18.745553335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:56.7073034 +0000 UTC m=+24.830893246" watchObservedRunningTime="2026-07-29 20:47:56.707848004 +0000 UTC m=+24.831437849" Jul 29 20:47:57.487393 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:57.487089 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:57.487609 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:57.487088 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:57.487609 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:57.487449 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:57.487609 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:57.487500 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:57.701117 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:57.701079 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" event={"ID":"d085f9bf-b95d-487e-953b-b20b22f4226d","Type":"ContainerStarted","Data":"356c450fd259ca84588de60b82a37b504a690e7ea18ea60a434105056be8a07b"} Jul 29 20:47:57.705675 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:57.705645 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" event={"ID":"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e","Type":"ContainerStarted","Data":"be6299243f668a91b10160cbc9dfb558181d2c5ee6361a87c2c48c97e2f72f95"} Jul 29 20:47:57.718500 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:57.718440 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x8qq" podStartSLOduration=4.749625566 podStartE2EDuration="25.718421335s" totalImagesPullingTime="20.968795769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:57.718256556 +0000 UTC m=+25.841846398" watchObservedRunningTime="2026-07-29 20:47:57.718421335 +0000 UTC m=+25.842011178" Jul 29 20:47:58.487369 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:58.487333 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:47:58.487648 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:58.487450 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:47:59.380435 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:59.380390 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v4k8h" Jul 29 20:47:59.381125 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:59.381105 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v4k8h" Jul 29 20:47:59.487426 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:59.487388 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:47:59.487616 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:59.487388 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:47:59.487616 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:59.487536 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:47:59.488113 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:47:59.487611 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:47:59.714705 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:59.714652 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" event={"ID":"e2ad47a9-1bdd-4d86-b0e1-fd05c2d4190e","Type":"ContainerStarted","Data":"f25a3a819ee918ded876ebf0f821494cb674411d84a40765689f5f3ee7be4caa"} Jul 29 20:47:59.716813 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:59.716790 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:59.717491 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:59.717120 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:59.717491 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:59.717191 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:59.717942 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:59.717522 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v4k8h" Jul 29 20:47:59.738540 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:59.738323 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:59.739418 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:59.738849 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:47:59.742954 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:47:59.742902 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" podStartSLOduration=8.917546298 podStartE2EDuration="27.742883007s" totalImagesPullingTime="18.825336709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:59.741352129 +0000 UTC m=+27.864941974" watchObservedRunningTime="2026-07-29 20:47:59.742883007 +0000 UTC m=+27.866472851" Jul 29 20:48:00.487246 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:00.487070 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:48:00.487396 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:00.487327 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:48:00.718906 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:00.718873 2582 generic.go:356] "Generic (PLEG): container finished" podID="44c3faed-67bf-41d8-b138-dd37c61977f6" containerID="2748d1a4e7ca75b4bd3331dbfef2243d5d1e032051f4f53ccff21d65a99c9fb1" exitCode=0 Jul 29 20:48:00.719528 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:00.718967 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxgzz" event={"ID":"44c3faed-67bf-41d8-b138-dd37c61977f6","Type":"ContainerDied","Data":"2748d1a4e7ca75b4bd3331dbfef2243d5d1e032051f4f53ccff21d65a99c9fb1"} Jul 29 20:48:01.486969 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:01.486941 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:48:01.487290 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:01.486976 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:48:01.487290 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:01.487085 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:48:01.487290 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:01.487201 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:48:01.885631 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:01.885442 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fqq45"] Jul 29 20:48:01.886046 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:01.885708 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:48:01.886046 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:01.885820 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:48:01.886182 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:01.886162 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c287f"] Jul 29 20:48:01.886268 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:01.886254 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:48:01.886389 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:01.886365 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:48:01.888191 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:01.888159 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-km6rd"] Jul 29 20:48:01.888284 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:01.888271 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:48:01.888402 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:01.888384 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:48:02.726126 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:02.726089 2582 generic.go:356] "Generic (PLEG): container finished" podID="44c3faed-67bf-41d8-b138-dd37c61977f6" containerID="6965dfc7bb23ede1f09ac66fc2cc788bad8dfa7a4c5168f180db6fa6b1c26c21" exitCode=0 Jul 29 20:48:02.726313 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:02.726150 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxgzz" event={"ID":"44c3faed-67bf-41d8-b138-dd37c61977f6","Type":"ContainerDied","Data":"6965dfc7bb23ede1f09ac66fc2cc788bad8dfa7a4c5168f180db6fa6b1c26c21"} Jul 29 20:48:03.487175 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:03.487143 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:48:03.487714 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:03.487185 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:48:03.487714 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:03.487185 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:48:03.487714 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:03.487279 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:48:03.487714 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:03.487344 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:48:03.487714 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:03.487424 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:48:04.732281 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:04.732249 2582 generic.go:356] "Generic (PLEG): container finished" podID="44c3faed-67bf-41d8-b138-dd37c61977f6" containerID="9560ad8e8470e28fa36d1bd29ec384f260ba9180a35cc0825e3edf2b79c9c585" exitCode=0 Jul 29 20:48:04.733070 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:04.732305 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxgzz" event={"ID":"44c3faed-67bf-41d8-b138-dd37c61977f6","Type":"ContainerDied","Data":"9560ad8e8470e28fa36d1bd29ec384f260ba9180a35cc0825e3edf2b79c9c585"} Jul 29 20:48:05.486876 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:05.486784 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:48:05.486876 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:05.486789 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:48:05.487128 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:05.486807 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:48:05.487128 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:05.486936 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:48:05.487128 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:05.487002 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fqq45" podUID="2c45ab5d-799b-4ffc-b8e1-40fc891d4d87" Jul 29 20:48:05.487128 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:05.487095 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c287f" podUID="cfd78950-23f5-4913-a500-8e477de213bd" Jul 29 20:48:06.140093 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:06.140057 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r72m\" (UniqueName: \"kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m\") pod \"network-check-target-c287f\" (UID: \"cfd78950-23f5-4913-a500-8e477de213bd\") " pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:48:06.140559 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:06.140122 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:48:06.140559 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:06.140158 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:48:06.140559 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:06.140235 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:48:06.140559 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:06.140260 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:48:06.140559 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:06.140273 2582 projected.go:196] Error preparing data for projected volume kube-api-access-2r72m for pod openshift-network-diagnostics/network-check-target-c287f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:48:06.140559 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:06.140289 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:48:06.140559 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:06.140323 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m podName:cfd78950-23f5-4913-a500-8e477de213bd nodeName:}" failed. No retries permitted until 2026-07-29 20:48:38.140309554 +0000 UTC m=+66.263899374 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2r72m" (UniqueName: "kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m") pod "network-check-target-c287f" (UID: "cfd78950-23f5-4913-a500-8e477de213bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:48:06.140559 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:06.140245 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:48:06.140559 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:06.140339 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret podName:2c45ab5d-799b-4ffc-b8e1-40fc891d4d87 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:38.140332762 +0000 UTC m=+66.263922582 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret") pod "global-pull-secret-syncer-fqq45" (UID: "2c45ab5d-799b-4ffc-b8e1-40fc891d4d87") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:48:06.140559 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:06.140363 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs podName:52b760b0-11c0-4512-aa57-fd441bdfd5a1 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:38.140347567 +0000 UTC m=+66.263937400 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs") pod "network-metrics-daemon-km6rd" (UID: "52b760b0-11c0-4512-aa57-fd441bdfd5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:48:07.151515 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.151486 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-160.ec2.internal" event="NodeReady" Jul 29 20:48:07.151973 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.151625 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 20:48:07.194449 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.194412 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vcr4b"] Jul 29 20:48:07.196937 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.196916 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.197085 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.197066 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wktqs"] Jul 29 20:48:07.199015 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.198993 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wktqs" Jul 29 20:48:07.199446 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.199427 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 20:48:07.199549 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.199535 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgjxr\"" Jul 29 20:48:07.199680 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.199663 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 20:48:07.201804 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.201789 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5tsgl\"" Jul 29 20:48:07.202056 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.202040 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 20:48:07.202145 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.202043 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 20:48:07.202276 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.202261 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 20:48:07.207692 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.207521 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vcr4b"] Jul 29 20:48:07.210283 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.210262 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wktqs"] Jul 29 20:48:07.346790 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.346709 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5858a6-49e8-4fc2-986a-e1d1e1658572-tmp-dir\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.346790 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.346746 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-724tq\" (UniqueName: \"kubernetes.io/projected/ea144d0b-7f3d-4833-a83b-c426404b2b12-kube-api-access-724tq\") pod \"ingress-canary-wktqs\" (UID: \"ea144d0b-7f3d-4833-a83b-c426404b2b12\") " pod="openshift-ingress-canary/ingress-canary-wktqs" Jul 29 20:48:07.346790 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.346787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.347052 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.346812 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea144d0b-7f3d-4833-a83b-c426404b2b12-cert\") pod \"ingress-canary-wktqs\" (UID: \"ea144d0b-7f3d-4833-a83b-c426404b2b12\") " pod="openshift-ingress-canary/ingress-canary-wktqs" Jul 29 20:48:07.347052 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.346841 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6j7\" (UniqueName: \"kubernetes.io/projected/bf5858a6-49e8-4fc2-986a-e1d1e1658572-kube-api-access-ht6j7\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.347052 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.346933 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5858a6-49e8-4fc2-986a-e1d1e1658572-config-volume\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.447550 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.447504 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea144d0b-7f3d-4833-a83b-c426404b2b12-cert\") pod \"ingress-canary-wktqs\" (UID: \"ea144d0b-7f3d-4833-a83b-c426404b2b12\") " pod="openshift-ingress-canary/ingress-canary-wktqs" Jul 29 20:48:07.447709 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.447561 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6j7\" (UniqueName: \"kubernetes.io/projected/bf5858a6-49e8-4fc2-986a-e1d1e1658572-kube-api-access-ht6j7\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.447709 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.447633 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5858a6-49e8-4fc2-986a-e1d1e1658572-config-volume\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.447709 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.447676 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5858a6-49e8-4fc2-986a-e1d1e1658572-tmp-dir\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.447709 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.447704 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-724tq\" (UniqueName: \"kubernetes.io/projected/ea144d0b-7f3d-4833-a83b-c426404b2b12-kube-api-access-724tq\") pod \"ingress-canary-wktqs\" (UID: \"ea144d0b-7f3d-4833-a83b-c426404b2b12\") " pod="openshift-ingress-canary/ingress-canary-wktqs" Jul 29 20:48:07.447933 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.447725 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.447933 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:07.447830 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:48:07.447933 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:07.447898 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls podName:bf5858a6-49e8-4fc2-986a-e1d1e1658572 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:07.94786729 +0000 UTC m=+36.071457115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls") pod "dns-default-vcr4b" (UID: "bf5858a6-49e8-4fc2-986a-e1d1e1658572") : secret "dns-default-metrics-tls" not found Jul 29 20:48:07.448347 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.448111 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5858a6-49e8-4fc2-986a-e1d1e1658572-tmp-dir\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.448486 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.448223 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5858a6-49e8-4fc2-986a-e1d1e1658572-config-volume\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.453735 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.453716 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea144d0b-7f3d-4833-a83b-c426404b2b12-cert\") pod \"ingress-canary-wktqs\" (UID: \"ea144d0b-7f3d-4833-a83b-c426404b2b12\") " pod="openshift-ingress-canary/ingress-canary-wktqs" Jul 29 20:48:07.456715 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.456697 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6j7\" (UniqueName: \"kubernetes.io/projected/bf5858a6-49e8-4fc2-986a-e1d1e1658572-kube-api-access-ht6j7\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.456902 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.456883 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-724tq\" (UniqueName: \"kubernetes.io/projected/ea144d0b-7f3d-4833-a83b-c426404b2b12-kube-api-access-724tq\") pod \"ingress-canary-wktqs\" (UID: \"ea144d0b-7f3d-4833-a83b-c426404b2b12\") " pod="openshift-ingress-canary/ingress-canary-wktqs" Jul 29 20:48:07.486740 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.486715 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:48:07.486740 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.486736 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:48:07.486870 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.486747 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:48:07.490315 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.490194 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 20:48:07.490315 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.490214 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vp87c\"" Jul 29 20:48:07.490315 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.490295 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 20:48:07.490970 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.490950 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 20:48:07.491077 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.491014 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 20:48:07.491077 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.491065 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ffmvq\"" Jul 29 20:48:07.515786 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.515769 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wktqs" Jul 29 20:48:07.952979 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:07.952938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:07.953171 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:07.953101 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:48:07.953243 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:07.953172 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls podName:bf5858a6-49e8-4fc2-986a-e1d1e1658572 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:08.953151675 +0000 UTC m=+37.076741509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls") pod "dns-default-vcr4b" (UID: "bf5858a6-49e8-4fc2-986a-e1d1e1658572") : secret "dns-default-metrics-tls" not found Jul 29 20:48:08.825303 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:08.825277 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wktqs"] Jul 29 20:48:08.828869 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:48:08.828847 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea144d0b_7f3d_4833_a83b_c426404b2b12.slice/crio-e81029aa75d555b901ddc5237d9f886628a2c650ba51aa4c4e9d6513482ce227 WatchSource:0}: Error finding container e81029aa75d555b901ddc5237d9f886628a2c650ba51aa4c4e9d6513482ce227: Status 404 returned error can't find the container with id e81029aa75d555b901ddc5237d9f886628a2c650ba51aa4c4e9d6513482ce227 Jul 29 20:48:08.962725 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:08.962658 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:08.962839 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:08.962788 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:48:08.962890 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:08.962857 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls podName:bf5858a6-49e8-4fc2-986a-e1d1e1658572 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:10.962836204 +0000 UTC m=+39.086426042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls") pod "dns-default-vcr4b" (UID: "bf5858a6-49e8-4fc2-986a-e1d1e1658572") : secret "dns-default-metrics-tls" not found Jul 29 20:48:09.747507 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:09.747478 2582 generic.go:356] "Generic (PLEG): container finished" podID="44c3faed-67bf-41d8-b138-dd37c61977f6" containerID="7191b8e916fbe5a0dc5b35db39a655a8daf46bb6891b025f04b1b5c3b81e475d" exitCode=0 Jul 29 20:48:09.747698 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:09.747561 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxgzz" event={"ID":"44c3faed-67bf-41d8-b138-dd37c61977f6","Type":"ContainerDied","Data":"7191b8e916fbe5a0dc5b35db39a655a8daf46bb6891b025f04b1b5c3b81e475d"} Jul 29 20:48:09.748885 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:09.748777 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wktqs" event={"ID":"ea144d0b-7f3d-4833-a83b-c426404b2b12","Type":"ContainerStarted","Data":"e81029aa75d555b901ddc5237d9f886628a2c650ba51aa4c4e9d6513482ce227"} Jul 29 20:48:10.755305 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:10.755056 2582 generic.go:356] "Generic (PLEG): container finished" podID="44c3faed-67bf-41d8-b138-dd37c61977f6" containerID="9405aa416b8cfb3a4984bdfedb7f2cd61169de30f88c4f66d5db2d8addf092ab" exitCode=0 Jul 29 20:48:10.755749 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:10.755133 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxgzz" event={"ID":"44c3faed-67bf-41d8-b138-dd37c61977f6","Type":"ContainerDied","Data":"9405aa416b8cfb3a4984bdfedb7f2cd61169de30f88c4f66d5db2d8addf092ab"} Jul 29 20:48:10.979712 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:10.979682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:10.979868 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:10.979847 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:48:10.979957 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:10.979945 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls podName:bf5858a6-49e8-4fc2-986a-e1d1e1658572 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:14.979922326 +0000 UTC m=+43.103512167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls") pod "dns-default-vcr4b" (UID: "bf5858a6-49e8-4fc2-986a-e1d1e1658572") : secret "dns-default-metrics-tls" not found Jul 29 20:48:11.760756 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:11.760720 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxgzz" event={"ID":"44c3faed-67bf-41d8-b138-dd37c61977f6","Type":"ContainerStarted","Data":"1d8d154b06daeb1601191e9396cb11f125c969b874bb9f5ff927c242efc1dab9"} Jul 29 20:48:11.765087 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:11.765047 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wktqs" event={"ID":"ea144d0b-7f3d-4833-a83b-c426404b2b12","Type":"ContainerStarted","Data":"44183c38af2313bdbbb181a18b4b13a49b7152aee4910f4c06df917be6a64ae5"} Jul 29 20:48:11.783954 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:11.783912 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sxgzz" podStartSLOduration=11.049421426 podStartE2EDuration="39.783895253s" totalImagesPullingTime="28.734473827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:48:11.782327517 +0000 UTC m=+39.905917361" watchObservedRunningTime="2026-07-29 20:48:11.783895253 +0000 UTC m=+39.907485096" Jul 29 20:48:11.797559 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:11.797513 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wktqs" podStartSLOduration=2.464496733 podStartE2EDuration="4.797499854s" totalImagesPullingTime="2.333003121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:48:11.797117202 +0000 UTC m=+39.920707046" watchObservedRunningTime="2026-07-29 20:48:11.797499854 +0000 UTC m=+39.921089696" Jul 29 20:48:15.008901 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:15.008858 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:15.009398 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:15.009005 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:48:15.009398 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:15.009099 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls podName:bf5858a6-49e8-4fc2-986a-e1d1e1658572 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:23.009077668 +0000 UTC m=+51.132667488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls") pod "dns-default-vcr4b" (UID: "bf5858a6-49e8-4fc2-986a-e1d1e1658572") : secret "dns-default-metrics-tls" not found Jul 29 20:48:23.064257 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:23.064215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:23.064714 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:23.064331 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:48:23.064714 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:23.064394 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls podName:bf5858a6-49e8-4fc2-986a-e1d1e1658572 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:39.064379954 +0000 UTC m=+67.187969774 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls") pod "dns-default-vcr4b" (UID: "bf5858a6-49e8-4fc2-986a-e1d1e1658572") : secret "dns-default-metrics-tls" not found Jul 29 20:48:31.732210 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:31.732177 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwjfw" Jul 29 20:48:38.234932 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.234880 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:48:38.234932 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.234943 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:48:38.235454 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.234974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r72m\" (UniqueName: \"kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m\") pod \"network-check-target-c287f\" (UID: \"cfd78950-23f5-4913-a500-8e477de213bd\") " pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:48:38.237666 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.237647 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 20:48:38.237714 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.237650 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 20:48:38.237714 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.237693 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 20:48:38.245847 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:38.245824 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 20:48:38.245955 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:38.245872 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs podName:52b760b0-11c0-4512-aa57-fd441bdfd5a1 nodeName:}" failed. No retries permitted until 2026-07-29 20:49:42.245857575 +0000 UTC m=+130.369447394 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs") pod "network-metrics-daemon-km6rd" (UID: "52b760b0-11c0-4512-aa57-fd441bdfd5a1") : secret "metrics-daemon-secret" not found Jul 29 20:48:38.247774 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.247757 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 20:48:38.249638 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.249620 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c45ab5d-799b-4ffc-b8e1-40fc891d4d87-original-pull-secret\") pod \"global-pull-secret-syncer-fqq45\" (UID: \"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87\") " pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:48:38.258952 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.258925 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r72m\" (UniqueName: \"kubernetes.io/projected/cfd78950-23f5-4913-a500-8e477de213bd-kube-api-access-2r72m\") pod \"network-check-target-c287f\" (UID: \"cfd78950-23f5-4913-a500-8e477de213bd\") " pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:48:38.404255 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.404220 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fqq45" Jul 29 20:48:38.411565 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.411539 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ffmvq\"" Jul 29 20:48:38.419814 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.419792 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:48:38.535536 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.535502 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fqq45"] Jul 29 20:48:38.537940 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:48:38.537912 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c45ab5d_799b_4ffc_b8e1_40fc891d4d87.slice/crio-1354ee47d8fc96cc77abfcc4708f115bdcf7a0f7fcba974aa2d5755250f9a60d WatchSource:0}: Error finding container 1354ee47d8fc96cc77abfcc4708f115bdcf7a0f7fcba974aa2d5755250f9a60d: Status 404 returned error can't find the container with id 1354ee47d8fc96cc77abfcc4708f115bdcf7a0f7fcba974aa2d5755250f9a60d Jul 29 20:48:38.547239 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.547218 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c287f"] Jul 29 20:48:38.549834 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:48:38.549814 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd78950_23f5_4913_a500_8e477de213bd.slice/crio-de517c73e2afdf6c86a4461d7745a988ca5437ec27e5b8d2eeb75955c3474b00 WatchSource:0}: Error finding container de517c73e2afdf6c86a4461d7745a988ca5437ec27e5b8d2eeb75955c3474b00: Status 404 returned error can't find the container with id de517c73e2afdf6c86a4461d7745a988ca5437ec27e5b8d2eeb75955c3474b00 Jul 29 20:48:38.831735 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.831706 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fqq45" event={"ID":"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87","Type":"ContainerStarted","Data":"1354ee47d8fc96cc77abfcc4708f115bdcf7a0f7fcba974aa2d5755250f9a60d"} Jul 29 20:48:38.832746 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:38.832726 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c287f" event={"ID":"cfd78950-23f5-4913-a500-8e477de213bd","Type":"ContainerStarted","Data":"de517c73e2afdf6c86a4461d7745a988ca5437ec27e5b8d2eeb75955c3474b00"} Jul 29 20:48:39.141663 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:39.141575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:48:39.141830 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:39.141758 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:48:39.141898 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:48:39.141837 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls podName:bf5858a6-49e8-4fc2-986a-e1d1e1658572 nodeName:}" failed. No retries permitted until 2026-07-29 20:49:11.14181878 +0000 UTC m=+99.265408600 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls") pod "dns-default-vcr4b" (UID: "bf5858a6-49e8-4fc2-986a-e1d1e1658572") : secret "dns-default-metrics-tls" not found Jul 29 20:48:43.847268 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:43.847232 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c287f" event={"ID":"cfd78950-23f5-4913-a500-8e477de213bd","Type":"ContainerStarted","Data":"36ee22d6b02867737f5714f97eee14bfcc19fbd169e4984ab40f278046f92ba2"} Jul 29 20:48:43.847819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:43.847347 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:48:43.851868 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:43.851839 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fqq45" event={"ID":"2c45ab5d-799b-4ffc-b8e1-40fc891d4d87","Type":"ContainerStarted","Data":"6c0648d4914397c817f6d7ce2b298e981746d21e7fffedba73cd79fb8c2c3419"} Jul 29 20:48:43.864368 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:43.864327 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c287f" podStartSLOduration=67.299505485 podStartE2EDuration="1m11.864314894s" totalImagesPullingTime="4.564809409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:48:43.863258587 +0000 UTC m=+71.986848426" watchObservedRunningTime="2026-07-29 20:48:43.864314894 +0000 UTC m=+71.987904741" Jul 29 20:48:43.876938 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:48:43.876890 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fqq45" podStartSLOduration=67.296995188 podStartE2EDuration="1m11.876878462s" totalImagesPullingTime="4.579883274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:48:43.876628823 +0000 UTC m=+72.000218669" watchObservedRunningTime="2026-07-29 20:48:43.876878462 +0000 UTC m=+72.000468306" Jul 29 20:49:11.153396 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:11.153270 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:49:11.153896 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:49:11.153415 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:49:11.153896 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:49:11.153487 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls podName:bf5858a6-49e8-4fc2-986a-e1d1e1658572 nodeName:}" failed. No retries permitted until 2026-07-29 20:50:15.153470735 +0000 UTC m=+163.277060558 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls") pod "dns-default-vcr4b" (UID: "bf5858a6-49e8-4fc2-986a-e1d1e1658572") : secret "dns-default-metrics-tls" not found Jul 29 20:49:14.856665 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:14.856637 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c287f" Jul 29 20:49:39.279559 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.279532 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62"] Jul 29 20:49:39.281470 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.281454 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:39.283988 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.283965 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 20:49:39.284128 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.283962 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 20:49:39.285073 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.285039 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 20:49:39.285073 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.285067 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 29 20:49:39.289413 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.289395 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj"] Jul 29 20:49:39.291421 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.291405 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.292659 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.292627 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62"] Jul 29 20:49:39.293879 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.293861 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 29 20:49:39.294001 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.293907 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 29 20:49:39.294001 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.293861 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 29 20:49:39.294145 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.294004 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 29 20:49:39.304554 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.304533 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj"] Jul 29 20:49:39.342875 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.340445 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae-klusterlet-config\") pod \"klusterlet-addon-workmgr-649fc77f4b-txm62\" (UID: \"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:39.342875 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.340532 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae-tmp\") pod \"klusterlet-addon-workmgr-649fc77f4b-txm62\" (UID: \"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:39.342875 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.340585 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhczp\" (UniqueName: \"kubernetes.io/projected/bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae-kube-api-access-dhczp\") pod \"klusterlet-addon-workmgr-649fc77f4b-txm62\" (UID: \"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:39.441438 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.441399 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt4dg\" (UniqueName: \"kubernetes.io/projected/3cd95744-248b-412e-9f90-b16baf2c17dd-kube-api-access-gt4dg\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.441530 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.441463 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd95744-248b-412e-9f90-b16baf2c17dd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.441530 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.441488 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae-klusterlet-config\") pod \"klusterlet-addon-workmgr-649fc77f4b-txm62\" (UID: \"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:39.441530 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.441503 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3cd95744-248b-412e-9f90-b16baf2c17dd-ca\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.441530 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.441529 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae-tmp\") pod \"klusterlet-addon-workmgr-649fc77f4b-txm62\" (UID: \"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:39.441704 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.441542 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3cd95744-248b-412e-9f90-b16baf2c17dd-hub\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.441704 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.441578 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cd95744-248b-412e-9f90-b16baf2c17dd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.441704 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.441603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhczp\" (UniqueName: \"kubernetes.io/projected/bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae-kube-api-access-dhczp\") pod \"klusterlet-addon-workmgr-649fc77f4b-txm62\" (UID: \"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:39.441841 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.441713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3cd95744-248b-412e-9f90-b16baf2c17dd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.441912 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.441893 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae-tmp\") pod \"klusterlet-addon-workmgr-649fc77f4b-txm62\" (UID: \"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:39.443990 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.443967 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae-klusterlet-config\") pod \"klusterlet-addon-workmgr-649fc77f4b-txm62\" (UID: \"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:39.450752 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.450731 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhczp\" (UniqueName: \"kubernetes.io/projected/bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae-kube-api-access-dhczp\") pod \"klusterlet-addon-workmgr-649fc77f4b-txm62\" (UID: \"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:39.542640 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.542565 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cd95744-248b-412e-9f90-b16baf2c17dd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.542640 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.542611 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3cd95744-248b-412e-9f90-b16baf2c17dd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.542640 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.542637 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt4dg\" (UniqueName: \"kubernetes.io/projected/3cd95744-248b-412e-9f90-b16baf2c17dd-kube-api-access-gt4dg\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.542811 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.542685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd95744-248b-412e-9f90-b16baf2c17dd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.542811 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.542714 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3cd95744-248b-412e-9f90-b16baf2c17dd-ca\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.542984 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.542954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3cd95744-248b-412e-9f90-b16baf2c17dd-hub\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.543467 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.543440 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd95744-248b-412e-9f90-b16baf2c17dd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.545162 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.545137 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3cd95744-248b-412e-9f90-b16baf2c17dd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.545266 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.545189 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3cd95744-248b-412e-9f90-b16baf2c17dd-ca\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.545336 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.545299 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cd95744-248b-412e-9f90-b16baf2c17dd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.545466 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.545445 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3cd95744-248b-412e-9f90-b16baf2c17dd-hub\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.551038 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.551001 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt4dg\" (UniqueName: \"kubernetes.io/projected/3cd95744-248b-412e-9f90-b16baf2c17dd-kube-api-access-gt4dg\") pod \"cluster-proxy-proxy-agent-5956b95cd4-xjqlj\" (UID: \"3cd95744-248b-412e-9f90-b16baf2c17dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.590968 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.590941 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:39.609759 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.609738 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:49:39.721717 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.721677 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62"] Jul 29 20:49:39.726262 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:49:39.726231 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2bc18f_2960_4ac8_ad2d_a1f45f9a13ae.slice/crio-d273fd2dd0056e9348d5c9315bfc62720105cb2d6a58af531c5f4ba09fd1bbb0 WatchSource:0}: Error finding container d273fd2dd0056e9348d5c9315bfc62720105cb2d6a58af531c5f4ba09fd1bbb0: Status 404 returned error can't find the container with id d273fd2dd0056e9348d5c9315bfc62720105cb2d6a58af531c5f4ba09fd1bbb0 Jul 29 20:49:39.742820 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.742802 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj"] Jul 29 20:49:39.744948 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:49:39.744925 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cd95744_248b_412e_9f90_b16baf2c17dd.slice/crio-fa4d21e01650fcc86c4bc6bbe62f6c7d4b1ac5b3faf5a5a53a397a4eae4acf5f WatchSource:0}: Error finding container fa4d21e01650fcc86c4bc6bbe62f6c7d4b1ac5b3faf5a5a53a397a4eae4acf5f: Status 404 returned error can't find the container with id fa4d21e01650fcc86c4bc6bbe62f6c7d4b1ac5b3faf5a5a53a397a4eae4acf5f Jul 29 20:49:39.984507 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.984435 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" event={"ID":"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae","Type":"ContainerStarted","Data":"d273fd2dd0056e9348d5c9315bfc62720105cb2d6a58af531c5f4ba09fd1bbb0"} Jul 29 20:49:39.985440 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:39.985419 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" event={"ID":"3cd95744-248b-412e-9f90-b16baf2c17dd","Type":"ContainerStarted","Data":"fa4d21e01650fcc86c4bc6bbe62f6c7d4b1ac5b3faf5a5a53a397a4eae4acf5f"} Jul 29 20:49:42.265833 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:42.265796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:49:42.266507 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:49:42.265968 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 20:49:42.266507 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:49:42.266084 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs podName:52b760b0-11c0-4512-aa57-fd441bdfd5a1 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:44.266059698 +0000 UTC m=+252.389649534 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs") pod "network-metrics-daemon-km6rd" (UID: "52b760b0-11c0-4512-aa57-fd441bdfd5a1") : secret "metrics-daemon-secret" not found Jul 29 20:49:46.006220 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:46.006179 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" event={"ID":"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae","Type":"ContainerStarted","Data":"114cf14fb85611ac8800bff1dcf3fd9939bdf15de22eef6961ef9a3486b61d03"} Jul 29 20:49:46.006699 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:46.006385 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:46.008270 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:46.008250 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:49:46.008496 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:46.008474 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" event={"ID":"3cd95744-248b-412e-9f90-b16baf2c17dd","Type":"ContainerStarted","Data":"d7f2515f1356e7e6593a9744c1ff6c1704ed248995decbb4abebb00ccb13a025"} Jul 29 20:49:46.008564 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:46.008510 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" event={"ID":"3cd95744-248b-412e-9f90-b16baf2c17dd","Type":"ContainerStarted","Data":"9148f926585e26cc08560516b23aea7f46f598cbd4e4733d7edb07298265ac45"} Jul 29 20:49:46.008564 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:46.008527 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" event={"ID":"3cd95744-248b-412e-9f90-b16baf2c17dd","Type":"ContainerStarted","Data":"f355566e80a8000e3aa4549738ef432c2647fa015e3432491232861a633e52b8"} Jul 29 20:49:46.022059 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:46.021999 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" podStartSLOduration=1.762662664 podStartE2EDuration="7.021985183s" totalImagesPullingTime="5.259322519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:49:46.021281126 +0000 UTC m=+134.144870969" watchObservedRunningTime="2026-07-29 20:49:46.021985183 +0000 UTC m=+134.145575025" Jul 29 20:49:46.064019 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:49:46.063966 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" podStartSLOduration=1.8240420309999998 podStartE2EDuration="7.063951406s" totalImagesPullingTime="5.239909375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:49:46.062636803 +0000 UTC m=+134.186226644" watchObservedRunningTime="2026-07-29 20:49:46.063951406 +0000 UTC m=+134.187541247" Jul 29 20:50:10.209424 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:50:10.209385 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vcr4b" podUID="bf5858a6-49e8-4fc2-986a-e1d1e1658572" Jul 29 20:50:10.497111 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:50:10.497075 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:50:11.076631 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:50:11.076599 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vcr4b" Jul 29 20:50:15.191805 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:50:15.191773 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:50:15.192308 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:50:15.191901 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:50:15.192308 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:50:15.191974 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls podName:bf5858a6-49e8-4fc2-986a-e1d1e1658572 nodeName:}" failed. No retries permitted until 2026-07-29 20:52:17.191954104 +0000 UTC m=+285.315543939 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls") pod "dns-default-vcr4b" (UID: "bf5858a6-49e8-4fc2-986a-e1d1e1658572") : secret "dns-default-metrics-tls" not found Jul 29 20:50:23.487116 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:50:23.487007 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:51:43.393254 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.393199 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hl6cv"] Jul 29 20:51:43.396292 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.396261 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.398797 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.398775 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 20:51:43.400043 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.399998 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 20:51:43.402586 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.402565 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 20:51:43.402795 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.402775 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9wqct\"" Jul 29 20:51:43.403051 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.403037 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 20:51:43.406421 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.406404 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hl6cv"] Jul 29 20:51:43.478001 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.477981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.478104 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.478009 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcpqz\" (UniqueName: \"kubernetes.io/projected/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-api-access-rcpqz\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.478104 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.478085 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75b76f3b-b444-47eb-bd66-67f5de1e2c08-crio-socket\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.478173 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.478106 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.478173 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.478163 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75b76f3b-b444-47eb-bd66-67f5de1e2c08-data-volume\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.578680 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.578652 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75b76f3b-b444-47eb-bd66-67f5de1e2c08-crio-socket\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.578680 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.578681 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.578853 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.578705 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75b76f3b-b444-47eb-bd66-67f5de1e2c08-data-volume\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.578853 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.578732 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.578853 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.578751 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcpqz\" (UniqueName: \"kubernetes.io/projected/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-api-access-rcpqz\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.578853 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.578774 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75b76f3b-b444-47eb-bd66-67f5de1e2c08-crio-socket\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.578853 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:43.578800 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 20:51:43.578853 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:43.578834 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 20:51:43.579053 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:43.578883 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm podName:75b76f3b-b444-47eb-bd66-67f5de1e2c08 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:44.078862654 +0000 UTC m=+252.202452499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hl6cv" (UID: "75b76f3b-b444-47eb-bd66-67f5de1e2c08") : configmap "kube-rbac-proxy" not found Jul 29 20:51:43.579053 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:43.578903 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls podName:75b76f3b-b444-47eb-bd66-67f5de1e2c08 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:44.078893587 +0000 UTC m=+252.202483410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl6cv" (UID: "75b76f3b-b444-47eb-bd66-67f5de1e2c08") : secret "insights-runtime-extractor-tls" not found Jul 29 20:51:43.579133 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.579056 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75b76f3b-b444-47eb-bd66-67f5de1e2c08-data-volume\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:43.588494 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:43.588471 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcpqz\" (UniqueName: \"kubernetes.io/projected/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-api-access-rcpqz\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:44.082552 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:44.082520 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:44.082706 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:44.082573 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:44.082706 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:44.082649 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 20:51:44.082706 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:44.082676 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 20:51:44.082804 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:44.082728 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm podName:75b76f3b-b444-47eb-bd66-67f5de1e2c08 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:45.082708869 +0000 UTC m=+253.206298691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hl6cv" (UID: "75b76f3b-b444-47eb-bd66-67f5de1e2c08") : configmap "kube-rbac-proxy" not found Jul 29 20:51:44.082804 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:44.082743 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls podName:75b76f3b-b444-47eb-bd66-67f5de1e2c08 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:45.082736564 +0000 UTC m=+253.206326384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl6cv" (UID: "75b76f3b-b444-47eb-bd66-67f5de1e2c08") : secret "insights-runtime-extractor-tls" not found Jul 29 20:51:44.284363 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:44.284315 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:51:44.284536 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:44.284463 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 20:51:44.284536 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:44.284528 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs podName:52b760b0-11c0-4512-aa57-fd441bdfd5a1 nodeName:}" failed. No retries permitted until 2026-07-29 20:53:46.284511358 +0000 UTC m=+374.408101179 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs") pod "network-metrics-daemon-km6rd" (UID: "52b760b0-11c0-4512-aa57-fd441bdfd5a1") : secret "metrics-daemon-secret" not found Jul 29 20:51:45.090741 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:45.090719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:45.091096 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:45.090766 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:45.091096 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:45.090835 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 20:51:45.091096 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:45.090892 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 20:51:45.091096 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:45.090896 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm podName:75b76f3b-b444-47eb-bd66-67f5de1e2c08 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:47.090880672 +0000 UTC m=+255.214470492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hl6cv" (UID: "75b76f3b-b444-47eb-bd66-67f5de1e2c08") : configmap "kube-rbac-proxy" not found Jul 29 20:51:45.091096 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:45.091000 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls podName:75b76f3b-b444-47eb-bd66-67f5de1e2c08 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:47.090981093 +0000 UTC m=+255.214570915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl6cv" (UID: "75b76f3b-b444-47eb-bd66-67f5de1e2c08") : secret "insights-runtime-extractor-tls" not found Jul 29 20:51:45.341480 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:45.341399 2582 generic.go:356] "Generic (PLEG): container finished" podID="bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae" containerID="114cf14fb85611ac8800bff1dcf3fd9939bdf15de22eef6961ef9a3486b61d03" exitCode=1 Jul 29 20:51:45.341480 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:45.341451 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" event={"ID":"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae","Type":"ContainerDied","Data":"114cf14fb85611ac8800bff1dcf3fd9939bdf15de22eef6961ef9a3486b61d03"} Jul 29 20:51:45.341745 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:45.341732 2582 scope.go:117] "RemoveContainer" containerID="114cf14fb85611ac8800bff1dcf3fd9939bdf15de22eef6961ef9a3486b61d03" Jul 29 20:51:45.578429 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:45.578402 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v5f5q_4987df3e-bae4-4828-a79f-2cfac383e2d5/dns-node-resolver/0.log" Jul 29 20:51:46.007120 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:46.007085 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:51:46.179162 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:46.179140 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ltkns_72e4f020-94dc-446c-9ff1-20a7b4a2f446/node-ca/0.log" Jul 29 20:51:46.346312 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:46.346237 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" event={"ID":"bc2bc18f-2960-4ac8-ad2d-a1f45f9a13ae","Type":"ContainerStarted","Data":"0c884e253db02eec9a24647d202fdda990735470a72736d4af63401fd6c04439"} Jul 29 20:51:46.346480 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:46.346469 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:51:46.347007 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:46.346987 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-649fc77f4b-txm62" Jul 29 20:51:47.105773 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:47.105728 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:47.105959 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:47.105791 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:47.105959 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:47.105848 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 20:51:47.105959 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:47.105903 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 20:51:47.105959 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:47.105912 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm podName:75b76f3b-b444-47eb-bd66-67f5de1e2c08 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:51.105896694 +0000 UTC m=+259.229486516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hl6cv" (UID: "75b76f3b-b444-47eb-bd66-67f5de1e2c08") : configmap "kube-rbac-proxy" not found Jul 29 20:51:47.105959 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:47.105943 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls podName:75b76f3b-b444-47eb-bd66-67f5de1e2c08 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:51.10593214 +0000 UTC m=+259.229521964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl6cv" (UID: "75b76f3b-b444-47eb-bd66-67f5de1e2c08") : secret "insights-runtime-extractor-tls" not found Jul 29 20:51:47.179434 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:47.179409 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wktqs_ea144d0b-7f3d-4833-a83b-c426404b2b12/serve-healthcheck-canary/0.log" Jul 29 20:51:51.136958 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:51.136920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:51.137446 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:51.136976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:51.137446 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:51.137060 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 20:51:51.137446 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:51.137113 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 20:51:51.137446 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:51.137128 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm podName:75b76f3b-b444-47eb-bd66-67f5de1e2c08 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:59.137109948 +0000 UTC m=+267.260699769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hl6cv" (UID: "75b76f3b-b444-47eb-bd66-67f5de1e2c08") : configmap "kube-rbac-proxy" not found Jul 29 20:51:51.137446 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:51:51.137157 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls podName:75b76f3b-b444-47eb-bd66-67f5de1e2c08 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:59.137145139 +0000 UTC m=+267.260734958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hl6cv" (UID: "75b76f3b-b444-47eb-bd66-67f5de1e2c08") : secret "insights-runtime-extractor-tls" not found Jul 29 20:51:59.193415 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:59.193340 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:59.193415 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:59.193391 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:59.193825 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:59.193804 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75b76f3b-b444-47eb-bd66-67f5de1e2c08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:59.195872 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:59.195854 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75b76f3b-b444-47eb-bd66-67f5de1e2c08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hl6cv\" (UID: \"75b76f3b-b444-47eb-bd66-67f5de1e2c08\") " pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:59.305571 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:59.305548 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hl6cv" Jul 29 20:51:59.427221 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:51:59.427188 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hl6cv"] Jul 29 20:51:59.430439 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:51:59.430416 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b76f3b_b444_47eb_bd66_67f5de1e2c08.slice/crio-e8643de972a55ff09613685902f9965a8b73729b9f1738c9949e1a664b771ed2 WatchSource:0}: Error finding container e8643de972a55ff09613685902f9965a8b73729b9f1738c9949e1a664b771ed2: Status 404 returned error can't find the container with id e8643de972a55ff09613685902f9965a8b73729b9f1738c9949e1a664b771ed2 Jul 29 20:52:00.384626 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:00.384573 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hl6cv" event={"ID":"75b76f3b-b444-47eb-bd66-67f5de1e2c08","Type":"ContainerStarted","Data":"e8643de972a55ff09613685902f9965a8b73729b9f1738c9949e1a664b771ed2"} Jul 29 20:52:04.400580 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:04.400549 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hl6cv" event={"ID":"75b76f3b-b444-47eb-bd66-67f5de1e2c08","Type":"ContainerStarted","Data":"f8b92ada5deb72601749d66689905d34e1a025384258209f5afbc254c202d2fa"} Jul 29 20:52:08.386433 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.386395 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c8c48fd5c-mhvt5"] Jul 29 20:52:08.389449 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.389430 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.392084 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.392061 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 20:52:08.392181 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.392128 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ws2kf\"" Jul 29 20:52:08.392917 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.392893 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 20:52:08.393014 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.392900 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 20:52:08.398546 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.398529 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 20:52:08.403385 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.403364 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c8c48fd5c-mhvt5"] Jul 29 20:52:08.416997 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.416969 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hl6cv" event={"ID":"75b76f3b-b444-47eb-bd66-67f5de1e2c08","Type":"ContainerStarted","Data":"3d6eb8bb2ef09f863ffbde079b4001bccb869f26763f9a9f76a30602b250dddf"} Jul 29 20:52:08.466886 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.466852 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98hks\" (UniqueName: \"kubernetes.io/projected/02b94d5a-97fb-4149-ad60-937e3e1f6555-kube-api-access-98hks\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.467060 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.466896 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02b94d5a-97fb-4149-ad60-937e3e1f6555-installation-pull-secrets\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.467060 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.466925 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b94d5a-97fb-4149-ad60-937e3e1f6555-registry-tls\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.467173 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.467080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02b94d5a-97fb-4149-ad60-937e3e1f6555-image-registry-private-configuration\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.467173 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.467123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02b94d5a-97fb-4149-ad60-937e3e1f6555-trusted-ca\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.467173 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.467149 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02b94d5a-97fb-4149-ad60-937e3e1f6555-bound-sa-token\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.467307 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.467172 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02b94d5a-97fb-4149-ad60-937e3e1f6555-registry-certificates\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.467307 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.467201 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02b94d5a-97fb-4149-ad60-937e3e1f6555-ca-trust-extracted\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.567887 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.567859 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02b94d5a-97fb-4149-ad60-937e3e1f6555-trusted-ca\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.568082 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.567902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02b94d5a-97fb-4149-ad60-937e3e1f6555-bound-sa-token\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.568082 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.567932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02b94d5a-97fb-4149-ad60-937e3e1f6555-registry-certificates\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.568082 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.567960 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02b94d5a-97fb-4149-ad60-937e3e1f6555-ca-trust-extracted\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.568082 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.567984 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98hks\" (UniqueName: \"kubernetes.io/projected/02b94d5a-97fb-4149-ad60-937e3e1f6555-kube-api-access-98hks\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.568082 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.568009 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02b94d5a-97fb-4149-ad60-937e3e1f6555-installation-pull-secrets\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.568082 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.568056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b94d5a-97fb-4149-ad60-937e3e1f6555-registry-tls\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.568395 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.568109 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02b94d5a-97fb-4149-ad60-937e3e1f6555-image-registry-private-configuration\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.568706 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.568675 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02b94d5a-97fb-4149-ad60-937e3e1f6555-ca-trust-extracted\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.568887 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.568863 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02b94d5a-97fb-4149-ad60-937e3e1f6555-registry-certificates\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.569192 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.569144 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02b94d5a-97fb-4149-ad60-937e3e1f6555-trusted-ca\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.571271 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.571198 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b94d5a-97fb-4149-ad60-937e3e1f6555-registry-tls\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.571373 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.571271 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02b94d5a-97fb-4149-ad60-937e3e1f6555-image-registry-private-configuration\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.571373 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.571335 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02b94d5a-97fb-4149-ad60-937e3e1f6555-installation-pull-secrets\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.576733 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.576709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02b94d5a-97fb-4149-ad60-937e3e1f6555-bound-sa-token\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.576824 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.576756 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98hks\" (UniqueName: \"kubernetes.io/projected/02b94d5a-97fb-4149-ad60-937e3e1f6555-kube-api-access-98hks\") pod \"image-registry-c8c48fd5c-mhvt5\" (UID: \"02b94d5a-97fb-4149-ad60-937e3e1f6555\") " pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.700984 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.700871 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:08.843317 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:08.843281 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c8c48fd5c-mhvt5"] Jul 29 20:52:08.846305 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:52:08.846266 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b94d5a_97fb_4149_ad60_937e3e1f6555.slice/crio-c1123c9a848cf7cd36437f98fff03c237c4d9e8f1a89327d9c33fe678dbc64f6 WatchSource:0}: Error finding container c1123c9a848cf7cd36437f98fff03c237c4d9e8f1a89327d9c33fe678dbc64f6: Status 404 returned error can't find the container with id c1123c9a848cf7cd36437f98fff03c237c4d9e8f1a89327d9c33fe678dbc64f6 Jul 29 20:52:09.423588 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:09.423550 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" event={"ID":"02b94d5a-97fb-4149-ad60-937e3e1f6555","Type":"ContainerStarted","Data":"93f65f7c1abd8d594ef456b3dcefab159bf59ef328e1a10a4f7a41d49595c537"} Jul 29 20:52:09.424052 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:09.423613 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" event={"ID":"02b94d5a-97fb-4149-ad60-937e3e1f6555","Type":"ContainerStarted","Data":"c1123c9a848cf7cd36437f98fff03c237c4d9e8f1a89327d9c33fe678dbc64f6"} Jul 29 20:52:09.424052 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:09.423805 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:09.445329 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:09.444934 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" podStartSLOduration=1.444907737 podStartE2EDuration="1.444907737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:09.4428614 +0000 UTC m=+277.566451243" watchObservedRunningTime="2026-07-29 20:52:09.444907737 +0000 UTC m=+277.568497584" Jul 29 20:52:11.432645 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:11.432555 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hl6cv" event={"ID":"75b76f3b-b444-47eb-bd66-67f5de1e2c08","Type":"ContainerStarted","Data":"bd49fd9c1e1538a9e479760714c0500299014c7b6f573db3a4fa20e293ba92c5"} Jul 29 20:52:11.453009 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:11.452965 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hl6cv" podStartSLOduration=16.913480286 podStartE2EDuration="28.45294925s" totalImagesPullingTime="11.539468964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:51:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:11.451238175 +0000 UTC m=+279.574828017" watchObservedRunningTime="2026-07-29 20:52:11.45294925 +0000 UTC m=+279.576539126" Jul 29 20:52:14.077087 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:52:14.077014 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vcr4b" podUID="bf5858a6-49e8-4fc2-986a-e1d1e1658572" Jul 29 20:52:14.440974 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:14.440895 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vcr4b" Jul 29 20:52:17.227504 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.227472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:52:17.229906 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.229883 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf5858a6-49e8-4fc2-986a-e1d1e1658572-metrics-tls\") pod \"dns-default-vcr4b\" (UID: \"bf5858a6-49e8-4fc2-986a-e1d1e1658572\") " pod="openshift-dns/dns-default-vcr4b" Jul 29 20:52:17.412424 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.412393 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2f4sz"] Jul 29 20:52:17.418488 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.418464 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.421145 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.421124 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 20:52:17.421269 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.421143 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 20:52:17.421952 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.421931 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 20:52:17.422080 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.421954 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 20:52:17.422080 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.421971 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 20:52:17.422080 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.422008 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 20:52:17.422080 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.421971 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5q2kp\"" Jul 29 20:52:17.443885 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.443868 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgjxr\"" Jul 29 20:52:17.452627 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.452593 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vcr4b" Jul 29 20:52:17.529797 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.529406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-sys\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.529797 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.529471 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-accelerators-collector-config\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.529797 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.529517 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-metrics-client-ca\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.529797 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.529558 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-textfile\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.529797 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.529618 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.529797 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.529657 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-wtmp\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.529797 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.529687 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxw8x\" (UniqueName: \"kubernetes.io/projected/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-kube-api-access-xxw8x\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.529797 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.529715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-tls\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.529797 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.529753 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-root\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.578330 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.578299 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vcr4b"] Jul 29 20:52:17.581354 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:52:17.581330 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5858a6_49e8_4fc2_986a_e1d1e1658572.slice/crio-be9dd3fb6045118d8f93aff02608ad0370bb3e57814e99391c940643b80ea61c WatchSource:0}: Error finding container be9dd3fb6045118d8f93aff02608ad0370bb3e57814e99391c940643b80ea61c: Status 404 returned error can't find the container with id be9dd3fb6045118d8f93aff02608ad0370bb3e57814e99391c940643b80ea61c Jul 29 20:52:17.630717 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.630683 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-wtmp\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.630872 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.630721 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxw8x\" (UniqueName: \"kubernetes.io/projected/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-kube-api-access-xxw8x\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.630872 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.630748 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-tls\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.630872 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.630776 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-root\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.630872 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.630815 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-sys\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.630872 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.630850 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-accelerators-collector-config\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.630872 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.630859 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-wtmp\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.631294 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.630888 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-metrics-client-ca\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.631294 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.630897 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-root\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.631294 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.630922 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-sys\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.631294 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.630926 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-textfile\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.631294 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:52:17.630927 2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 20:52:17.631294 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.630958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.631294 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:52:17.631012 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-tls podName:7d8f5523-e7fb-44e6-8ac2-271a16cf4443 nodeName:}" failed. No retries permitted until 2026-07-29 20:52:18.130988117 +0000 UTC m=+286.254577944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-tls") pod "node-exporter-2f4sz" (UID: "7d8f5523-e7fb-44e6-8ac2-271a16cf4443") : secret "node-exporter-tls" not found Jul 29 20:52:17.631294 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.631275 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-textfile\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.631602 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.631493 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-accelerators-collector-config\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.631602 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.631513 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-metrics-client-ca\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.633407 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.633382 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:17.641798 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:17.641775 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxw8x\" (UniqueName: \"kubernetes.io/projected/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-kube-api-access-xxw8x\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:18.135291 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:18.135262 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-tls\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:18.137840 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:18.137811 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d8f5523-e7fb-44e6-8ac2-271a16cf4443-node-exporter-tls\") pod \"node-exporter-2f4sz\" (UID: \"7d8f5523-e7fb-44e6-8ac2-271a16cf4443\") " pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:18.327975 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:18.327941 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2f4sz" Jul 29 20:52:18.339518 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:52:18.339488 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8f5523_e7fb_44e6_8ac2_271a16cf4443.slice/crio-dad10b60c3d8bae8e3c115ba9d326bd8ff6ccb47fefe0ed34bbce379680491fc WatchSource:0}: Error finding container dad10b60c3d8bae8e3c115ba9d326bd8ff6ccb47fefe0ed34bbce379680491fc: Status 404 returned error can't find the container with id dad10b60c3d8bae8e3c115ba9d326bd8ff6ccb47fefe0ed34bbce379680491fc Jul 29 20:52:18.454746 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:18.454671 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vcr4b" event={"ID":"bf5858a6-49e8-4fc2-986a-e1d1e1658572","Type":"ContainerStarted","Data":"be9dd3fb6045118d8f93aff02608ad0370bb3e57814e99391c940643b80ea61c"} Jul 29 20:52:18.455666 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:18.455645 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f4sz" event={"ID":"7d8f5523-e7fb-44e6-8ac2-271a16cf4443","Type":"ContainerStarted","Data":"dad10b60c3d8bae8e3c115ba9d326bd8ff6ccb47fefe0ed34bbce379680491fc"} Jul 29 20:52:19.461788 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:19.461706 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vcr4b" event={"ID":"bf5858a6-49e8-4fc2-986a-e1d1e1658572","Type":"ContainerStarted","Data":"146a5a413b457d00e63a7a2a24e753d507021a973b769bbe36c0e93e6ae71fb3"} Jul 29 20:52:19.461788 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:19.461765 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vcr4b" event={"ID":"bf5858a6-49e8-4fc2-986a-e1d1e1658572","Type":"ContainerStarted","Data":"f9ee6b473ad1320d99dcd85e5feef9ddea5fa42384747d6eb1c7710b6e1f1974"} Jul 29 20:52:19.462211 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:19.461819 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vcr4b" Jul 29 20:52:19.480891 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:19.480845 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vcr4b" podStartSLOduration=250.949032336 podStartE2EDuration="4m12.480830698s" totalImagesPullingTime="1.531798362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:19.478705823 +0000 UTC m=+287.602295677" watchObservedRunningTime="2026-07-29 20:52:19.480830698 +0000 UTC m=+287.604420540" Jul 29 20:52:20.466165 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:20.466128 2582 generic.go:356] "Generic (PLEG): container finished" podID="7d8f5523-e7fb-44e6-8ac2-271a16cf4443" containerID="903cfeb12ddb729684a3d66c5e24aa25c7443c0c7c8221c59bec7157e6861ffe" exitCode=0 Jul 29 20:52:20.466551 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:20.466213 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f4sz" event={"ID":"7d8f5523-e7fb-44e6-8ac2-271a16cf4443","Type":"ContainerDied","Data":"903cfeb12ddb729684a3d66c5e24aa25c7443c0c7c8221c59bec7157e6861ffe"} Jul 29 20:52:21.471350 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.471313 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f4sz" event={"ID":"7d8f5523-e7fb-44e6-8ac2-271a16cf4443","Type":"ContainerStarted","Data":"e97dd860f6c11f4759ad617b372008a2d8123e0ec1d2f5dcb25b19b321b7b1d9"} Jul 29 20:52:21.471799 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.471371 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2f4sz" event={"ID":"7d8f5523-e7fb-44e6-8ac2-271a16cf4443","Type":"ContainerStarted","Data":"ccf3b7c901137ce836749443fbe3073050da8f25e976339be260a15ad932bcd2"} Jul 29 20:52:21.491501 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.491455 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2f4sz" podStartSLOduration=3.067904212 podStartE2EDuration="4.491438875s" totalImagesPullingTime="1.353786743s" totalInitContainerRuntime="69.74792ms" isStatefulPod=true podCreationTimestamp="2026-07-29 20:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:21.490452687 +0000 UTC m=+289.614042526" watchObservedRunningTime="2026-07-29 20:52:21.491438875 +0000 UTC m=+289.615028716" Jul 29 20:52:21.691310 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.691276 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-54ff6744d8-bbw8d"] Jul 29 20:52:21.705318 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.705291 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54ff6744d8-bbw8d"] Jul 29 20:52:21.705471 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.705398 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.708593 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.708566 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 29 20:52:21.708742 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.708632 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 29 20:52:21.708742 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.708686 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 20:52:21.708742 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.708686 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-71tlpnoc8r9nr\"" Jul 29 20:52:21.708895 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.708846 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vvfvc\"" Jul 29 20:52:21.708961 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.708926 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 29 20:52:21.763737 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.763719 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b44b094a-135d-4683-a0d4-a2044b2ab112-audit-log\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.763833 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.763749 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b44b094a-135d-4683-a0d4-a2044b2ab112-metrics-server-audit-profiles\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.763833 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.763779 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b44b094a-135d-4683-a0d4-a2044b2ab112-secret-metrics-server-tls\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.763943 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.763873 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b44b094a-135d-4683-a0d4-a2044b2ab112-secret-metrics-server-client-certs\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.763943 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.763912 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b44b094a-135d-4683-a0d4-a2044b2ab112-client-ca-bundle\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.764003 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.763946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44b094a-135d-4683-a0d4-a2044b2ab112-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.764003 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.763967 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5pxt\" (UniqueName: \"kubernetes.io/projected/b44b094a-135d-4683-a0d4-a2044b2ab112-kube-api-access-p5pxt\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.864585 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.864561 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b44b094a-135d-4683-a0d4-a2044b2ab112-client-ca-bundle\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.864708 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.864602 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44b094a-135d-4683-a0d4-a2044b2ab112-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.864708 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.864629 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5pxt\" (UniqueName: \"kubernetes.io/projected/b44b094a-135d-4683-a0d4-a2044b2ab112-kube-api-access-p5pxt\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.864708 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.864649 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b44b094a-135d-4683-a0d4-a2044b2ab112-audit-log\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.864708 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.864677 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b44b094a-135d-4683-a0d4-a2044b2ab112-metrics-server-audit-profiles\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.864708 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.864706 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b44b094a-135d-4683-a0d4-a2044b2ab112-secret-metrics-server-tls\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.864943 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.864749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b44b094a-135d-4683-a0d4-a2044b2ab112-secret-metrics-server-client-certs\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.865260 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.865233 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b44b094a-135d-4683-a0d4-a2044b2ab112-audit-log\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.865479 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.865460 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44b094a-135d-4683-a0d4-a2044b2ab112-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.866087 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.866063 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b44b094a-135d-4683-a0d4-a2044b2ab112-metrics-server-audit-profiles\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.867252 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.867230 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b44b094a-135d-4683-a0d4-a2044b2ab112-secret-metrics-server-tls\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.867345 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.867319 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b44b094a-135d-4683-a0d4-a2044b2ab112-client-ca-bundle\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.867419 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.867394 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b44b094a-135d-4683-a0d4-a2044b2ab112-secret-metrics-server-client-certs\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:21.873251 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:21.873230 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5pxt\" (UniqueName: \"kubernetes.io/projected/b44b094a-135d-4683-a0d4-a2044b2ab112-kube-api-access-p5pxt\") pod \"metrics-server-54ff6744d8-bbw8d\" (UID: \"b44b094a-135d-4683-a0d4-a2044b2ab112\") " pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:22.014994 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:22.014927 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:22.201629 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:22.201588 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54ff6744d8-bbw8d"] Jul 29 20:52:22.206406 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:52:22.206369 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44b094a_135d_4683_a0d4_a2044b2ab112.slice/crio-31a9ce3402c1f5fbabc7e5a35726cc59ef70b3c64c617979789c8665e2078b61 WatchSource:0}: Error finding container 31a9ce3402c1f5fbabc7e5a35726cc59ef70b3c64c617979789c8665e2078b61: Status 404 returned error can't find the container with id 31a9ce3402c1f5fbabc7e5a35726cc59ef70b3c64c617979789c8665e2078b61 Jul 29 20:52:22.476754 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:22.476672 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" event={"ID":"b44b094a-135d-4683-a0d4-a2044b2ab112","Type":"ContainerStarted","Data":"31a9ce3402c1f5fbabc7e5a35726cc59ef70b3c64c617979789c8665e2078b61"} Jul 29 20:52:23.823117 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.823085 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b77f4479d-6nvtd"] Jul 29 20:52:23.842656 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.842631 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b77f4479d-6nvtd"] Jul 29 20:52:23.842790 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.842739 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.845286 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.845260 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 20:52:23.845437 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.845341 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 20:52:23.845506 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.845436 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fbn88\"" Jul 29 20:52:23.846429 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.846368 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 20:52:23.846429 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.846386 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 20:52:23.846429 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.846412 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 20:52:23.846663 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.846518 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 20:52:23.846881 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.846862 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 20:52:23.850337 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.850322 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 20:52:23.882250 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.882226 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b558243-4ba4-44ca-9b78-638a1334e45a-console-serving-cert\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.882351 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.882279 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdn2x\" (UniqueName: \"kubernetes.io/projected/6b558243-4ba4-44ca-9b78-638a1334e45a-kube-api-access-gdn2x\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.882351 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.882334 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-console-config\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.882461 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.882375 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-service-ca\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.882461 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.882417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b558243-4ba4-44ca-9b78-638a1334e45a-console-oauth-config\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.882524 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.882474 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-oauth-serving-cert\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.882524 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.882513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-trusted-ca-bundle\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.983110 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.983083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdn2x\" (UniqueName: \"kubernetes.io/projected/6b558243-4ba4-44ca-9b78-638a1334e45a-kube-api-access-gdn2x\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.983233 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.983120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-console-config\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.983233 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.983136 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-service-ca\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.983233 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.983168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b558243-4ba4-44ca-9b78-638a1334e45a-console-oauth-config\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.983233 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.983199 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-oauth-serving-cert\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.983233 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.983221 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-trusted-ca-bundle\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.983474 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.983258 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b558243-4ba4-44ca-9b78-638a1334e45a-console-serving-cert\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.983927 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.983875 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-console-config\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.984085 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.984061 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-oauth-serving-cert\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.984174 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.984098 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-service-ca\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.984174 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.984163 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-trusted-ca-bundle\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.985706 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.985688 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b558243-4ba4-44ca-9b78-638a1334e45a-console-oauth-config\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.985841 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.985824 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b558243-4ba4-44ca-9b78-638a1334e45a-console-serving-cert\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:23.990564 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:23.990543 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdn2x\" (UniqueName: \"kubernetes.io/projected/6b558243-4ba4-44ca-9b78-638a1334e45a-kube-api-access-gdn2x\") pod \"console-7b77f4479d-6nvtd\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:24.151826 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:24.151802 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:24.289340 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:24.289313 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b77f4479d-6nvtd"] Jul 29 20:52:24.291616 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:52:24.291592 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b558243_4ba4_44ca_9b78_638a1334e45a.slice/crio-1559be008beab3f6532d63effe31e0f700042b2030d12b128bcc78e6721fa829 WatchSource:0}: Error finding container 1559be008beab3f6532d63effe31e0f700042b2030d12b128bcc78e6721fa829: Status 404 returned error can't find the container with id 1559be008beab3f6532d63effe31e0f700042b2030d12b128bcc78e6721fa829 Jul 29 20:52:24.486114 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:24.486011 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" event={"ID":"b44b094a-135d-4683-a0d4-a2044b2ab112","Type":"ContainerStarted","Data":"032f15b6b6561413cc514afbc37701db72a3a4078857eff01abdc94a079c07d3"} Jul 29 20:52:24.494110 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:24.491892 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b77f4479d-6nvtd" event={"ID":"6b558243-4ba4-44ca-9b78-638a1334e45a","Type":"ContainerStarted","Data":"1559be008beab3f6532d63effe31e0f700042b2030d12b128bcc78e6721fa829"} Jul 29 20:52:24.506977 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:24.506925 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" podStartSLOduration=1.6068484060000001 podStartE2EDuration="3.506908602s" totalImagesPullingTime="1.900060196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:24.505995495 +0000 UTC m=+292.629585348" watchObservedRunningTime="2026-07-29 20:52:24.506908602 +0000 UTC m=+292.630498445" Jul 29 20:52:26.487620 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:52:26.487577 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-km6rd" podUID="52b760b0-11c0-4512-aa57-fd441bdfd5a1" Jul 29 20:52:28.502096 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:28.502064 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b77f4479d-6nvtd" event={"ID":"6b558243-4ba4-44ca-9b78-638a1334e45a","Type":"ContainerStarted","Data":"6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f"} Jul 29 20:52:28.521859 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:28.521810 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b77f4479d-6nvtd" podStartSLOduration=2.215996938 podStartE2EDuration="5.521794867s" totalImagesPullingTime="3.305797929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:28.520011483 +0000 UTC m=+296.643601325" watchObservedRunningTime="2026-07-29 20:52:28.521794867 +0000 UTC m=+296.645384708" Jul 29 20:52:28.705662 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:28.705633 2582 patch_prober.go:28] interesting pod/image-registry-c8c48fd5c-mhvt5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 20:52:28.705820 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:28.705678 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" podUID="02b94d5a-97fb-4149-ad60-937e3e1f6555" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 20:52:29.467982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:29.467955 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vcr4b" Jul 29 20:52:29.611315 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:29.611283 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" podUID="3cd95744-248b-412e-9f90-b16baf2c17dd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 29 20:52:30.433171 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.433137 2582 patch_prober.go:28] interesting pod/image-registry-c8c48fd5c-mhvt5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 20:52:30.433372 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.433196 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" podUID="02b94d5a-97fb-4149-ad60-937e3e1f6555" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 20:52:30.688565 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.688492 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-755d7586fd-nhdhj"] Jul 29 20:52:30.690737 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.690722 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.707546 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.707520 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-755d7586fd-nhdhj"] Jul 29 20:52:30.740715 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.740687 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-trusted-ca-bundle\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.740715 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.740722 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-oauth-serving-cert\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.740905 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.740761 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg7zp\" (UniqueName: \"kubernetes.io/projected/91887504-5ac3-40e4-b834-e4f4fb2e3c20-kube-api-access-vg7zp\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.740905 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.740820 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-oauth-config\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.740905 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.740839 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-service-ca\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.740905 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.740861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-config\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.741094 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.740919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-serving-cert\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.841833 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.841806 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg7zp\" (UniqueName: \"kubernetes.io/projected/91887504-5ac3-40e4-b834-e4f4fb2e3c20-kube-api-access-vg7zp\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.841998 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.841852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-oauth-config\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.841998 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.841874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-service-ca\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.841998 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.841906 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-config\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.841998 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.841935 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-serving-cert\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.841998 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.841972 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-trusted-ca-bundle\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.842295 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.842125 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-oauth-serving-cert\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.842687 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.842618 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-service-ca\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.842898 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.842764 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-oauth-serving-cert\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.842898 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.842764 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-config\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.843102 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.843080 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-trusted-ca-bundle\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.844523 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.844504 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-oauth-config\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.844756 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.844728 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-serving-cert\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.851423 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.851407 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg7zp\" (UniqueName: \"kubernetes.io/projected/91887504-5ac3-40e4-b834-e4f4fb2e3c20-kube-api-access-vg7zp\") pod \"console-755d7586fd-nhdhj\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:30.999625 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:30.999597 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:31.135994 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:31.135957 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-755d7586fd-nhdhj"] Jul 29 20:52:31.138854 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:52:31.138830 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91887504_5ac3_40e4_b834_e4f4fb2e3c20.slice/crio-9a9b6ba900f7c48798fc23c6a18fde6a94ffd3f3db76fbf5e88bd10f46fca506 WatchSource:0}: Error finding container 9a9b6ba900f7c48798fc23c6a18fde6a94ffd3f3db76fbf5e88bd10f46fca506: Status 404 returned error can't find the container with id 9a9b6ba900f7c48798fc23c6a18fde6a94ffd3f3db76fbf5e88bd10f46fca506 Jul 29 20:52:31.513402 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:31.513368 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755d7586fd-nhdhj" event={"ID":"91887504-5ac3-40e4-b834-e4f4fb2e3c20","Type":"ContainerStarted","Data":"2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507"} Jul 29 20:52:31.513549 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:31.513416 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755d7586fd-nhdhj" event={"ID":"91887504-5ac3-40e4-b834-e4f4fb2e3c20","Type":"ContainerStarted","Data":"9a9b6ba900f7c48798fc23c6a18fde6a94ffd3f3db76fbf5e88bd10f46fca506"} Jul 29 20:52:31.532531 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:31.532454 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-755d7586fd-nhdhj" podStartSLOduration=1.532440552 podStartE2EDuration="1.532440552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:31.531383636 +0000 UTC m=+299.654973477" watchObservedRunningTime="2026-07-29 20:52:31.532440552 +0000 UTC m=+299.656030394" Jul 29 20:52:32.299310 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:32.299284 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 20:52:34.152593 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:34.152555 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:34.154128 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:34.152640 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:34.157518 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:34.157489 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:34.527327 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:34.527294 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:52:38.705560 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:38.705531 2582 patch_prober.go:28] interesting pod/image-registry-c8c48fd5c-mhvt5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 20:52:38.705917 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:38.705578 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" podUID="02b94d5a-97fb-4149-ad60-937e3e1f6555" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 20:52:39.610826 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:39.610791 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" podUID="3cd95744-248b-412e-9f90-b16baf2c17dd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 29 20:52:40.432013 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:40.431978 2582 patch_prober.go:28] interesting pod/image-registry-c8c48fd5c-mhvt5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 20:52:40.432417 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:40.432048 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" podUID="02b94d5a-97fb-4149-ad60-937e3e1f6555" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 20:52:41.000383 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:41.000345 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:41.000582 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:41.000501 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:41.005204 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:41.005183 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:41.487084 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:41.486985 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:52:41.547452 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:41.547429 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:52:41.594610 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:41.594578 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b77f4479d-6nvtd"] Jul 29 20:52:42.015197 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:42.015151 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:42.015377 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:42.015226 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:52:48.705717 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:48.705690 2582 patch_prober.go:28] interesting pod/image-registry-c8c48fd5c-mhvt5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 20:52:48.706193 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:48.705740 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" podUID="02b94d5a-97fb-4149-ad60-937e3e1f6555" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 20:52:48.706193 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:48.705776 2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:52:48.706193 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:48.706157 2582 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"93f65f7c1abd8d594ef456b3dcefab159bf59ef328e1a10a4f7a41d49595c537"} pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" containerMessage="Container registry failed liveness probe, will be restarted" Jul 29 20:52:48.710088 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:48.710063 2582 patch_prober.go:28] interesting pod/image-registry-c8c48fd5c-mhvt5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 20:52:48.710190 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:48.710116 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" podUID="02b94d5a-97fb-4149-ad60-937e3e1f6555" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 20:52:49.610883 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:49.610848 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" podUID="3cd95744-248b-412e-9f90-b16baf2c17dd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 29 20:52:49.611082 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:49.610916 2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" Jul 29 20:52:49.611371 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:49.611352 2582 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"d7f2515f1356e7e6593a9744c1ff6c1704ed248995decbb4abebb00ccb13a025"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 29 20:52:49.611442 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:49.611389 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" podUID="3cd95744-248b-412e-9f90-b16baf2c17dd" containerName="service-proxy" containerID="cri-o://d7f2515f1356e7e6593a9744c1ff6c1704ed248995decbb4abebb00ccb13a025" gracePeriod=30 Jul 29 20:52:50.574487 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:50.574457 2582 generic.go:356] "Generic (PLEG): container finished" podID="3cd95744-248b-412e-9f90-b16baf2c17dd" containerID="d7f2515f1356e7e6593a9744c1ff6c1704ed248995decbb4abebb00ccb13a025" exitCode=2 Jul 29 20:52:50.574898 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:50.574511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" event={"ID":"3cd95744-248b-412e-9f90-b16baf2c17dd","Type":"ContainerDied","Data":"d7f2515f1356e7e6593a9744c1ff6c1704ed248995decbb4abebb00ccb13a025"} Jul 29 20:52:50.574898 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:50.574545 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5956b95cd4-xjqlj" event={"ID":"3cd95744-248b-412e-9f90-b16baf2c17dd","Type":"ContainerStarted","Data":"5e366e40eea65afde832d28ea07ffab9745389217b2a76e18aa9f4ea972ee24a"} Jul 29 20:52:58.710679 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:58.710648 2582 patch_prober.go:28] interesting pod/image-registry-c8c48fd5c-mhvt5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 20:52:58.711139 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:52:58.710698 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" podUID="02b94d5a-97fb-4149-ad60-937e3e1f6555" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 20:53:02.021089 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:02.021060 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:53:02.026227 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:02.026202 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-54ff6744d8-bbw8d" Jul 29 20:53:06.613300 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.613263 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b77f4479d-6nvtd" podUID="6b558243-4ba4-44ca-9b78-638a1334e45a" containerName="console" containerID="cri-o://6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f" gracePeriod=15 Jul 29 20:53:06.843456 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.843435 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:53:06.928051 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.927969 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-service-ca\") pod \"6b558243-4ba4-44ca-9b78-638a1334e45a\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " Jul 29 20:53:06.928051 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.928007 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-console-config\") pod \"6b558243-4ba4-44ca-9b78-638a1334e45a\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " Jul 29 20:53:06.928051 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.928055 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-oauth-serving-cert\") pod \"6b558243-4ba4-44ca-9b78-638a1334e45a\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " Jul 29 20:53:06.928281 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.928078 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b558243-4ba4-44ca-9b78-638a1334e45a-console-serving-cert\") pod \"6b558243-4ba4-44ca-9b78-638a1334e45a\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " Jul 29 20:53:06.928281 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.928102 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-trusted-ca-bundle\") pod \"6b558243-4ba4-44ca-9b78-638a1334e45a\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " Jul 29 20:53:06.928281 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.928139 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdn2x\" (UniqueName: \"kubernetes.io/projected/6b558243-4ba4-44ca-9b78-638a1334e45a-kube-api-access-gdn2x\") pod \"6b558243-4ba4-44ca-9b78-638a1334e45a\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " Jul 29 20:53:06.928281 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.928164 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b558243-4ba4-44ca-9b78-638a1334e45a-console-oauth-config\") pod \"6b558243-4ba4-44ca-9b78-638a1334e45a\" (UID: \"6b558243-4ba4-44ca-9b78-638a1334e45a\") " Jul 29 20:53:06.928475 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.928426 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-service-ca" (OuterVolumeSpecName: "service-ca") pod "6b558243-4ba4-44ca-9b78-638a1334e45a" (UID: "6b558243-4ba4-44ca-9b78-638a1334e45a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:53:06.928529 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.928510 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6b558243-4ba4-44ca-9b78-638a1334e45a" (UID: "6b558243-4ba4-44ca-9b78-638a1334e45a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:53:06.928580 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.928530 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-console-config" (OuterVolumeSpecName: "console-config") pod "6b558243-4ba4-44ca-9b78-638a1334e45a" (UID: "6b558243-4ba4-44ca-9b78-638a1334e45a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:53:06.928580 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.928536 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6b558243-4ba4-44ca-9b78-638a1334e45a" (UID: "6b558243-4ba4-44ca-9b78-638a1334e45a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:53:06.930430 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.930403 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b558243-4ba4-44ca-9b78-638a1334e45a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6b558243-4ba4-44ca-9b78-638a1334e45a" (UID: "6b558243-4ba4-44ca-9b78-638a1334e45a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:53:06.930554 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.930534 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b558243-4ba4-44ca-9b78-638a1334e45a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6b558243-4ba4-44ca-9b78-638a1334e45a" (UID: "6b558243-4ba4-44ca-9b78-638a1334e45a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:53:06.930615 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:06.930532 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b558243-4ba4-44ca-9b78-638a1334e45a-kube-api-access-gdn2x" (OuterVolumeSpecName: "kube-api-access-gdn2x") pod "6b558243-4ba4-44ca-9b78-638a1334e45a" (UID: "6b558243-4ba4-44ca-9b78-638a1334e45a"). InnerVolumeSpecName "kube-api-access-gdn2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:53:07.028655 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.028622 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-service-ca\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:53:07.028655 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.028651 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-console-config\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:53:07.028655 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.028659 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-oauth-serving-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:53:07.028818 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.028669 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b558243-4ba4-44ca-9b78-638a1334e45a-console-serving-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:53:07.028818 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.028677 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b558243-4ba4-44ca-9b78-638a1334e45a-trusted-ca-bundle\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:53:07.028818 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.028685 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdn2x\" (UniqueName: \"kubernetes.io/projected/6b558243-4ba4-44ca-9b78-638a1334e45a-kube-api-access-gdn2x\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:53:07.028818 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.028694 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b558243-4ba4-44ca-9b78-638a1334e45a-console-oauth-config\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:53:07.632079 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.632043 2582 generic.go:356] "Generic (PLEG): container finished" podID="6b558243-4ba4-44ca-9b78-638a1334e45a" containerID="6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f" exitCode=0 Jul 29 20:53:07.632079 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.632082 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b77f4479d-6nvtd" Jul 29 20:53:07.632555 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.632116 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b77f4479d-6nvtd" event={"ID":"6b558243-4ba4-44ca-9b78-638a1334e45a","Type":"ContainerDied","Data":"6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f"} Jul 29 20:53:07.632555 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.632166 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b77f4479d-6nvtd" event={"ID":"6b558243-4ba4-44ca-9b78-638a1334e45a","Type":"ContainerDied","Data":"1559be008beab3f6532d63effe31e0f700042b2030d12b128bcc78e6721fa829"} Jul 29 20:53:07.632555 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.632208 2582 scope.go:117] "RemoveContainer" containerID="6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f" Jul 29 20:53:07.640959 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.640938 2582 scope.go:117] "RemoveContainer" containerID="6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f" Jul 29 20:53:07.641251 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:53:07.641232 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f\": container with ID starting with 6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f not found: ID does not exist" containerID="6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f" Jul 29 20:53:07.641305 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.641260 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f"} err="failed to get container status \"6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f\": rpc error: code = NotFound desc = could not find container \"6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f\": container with ID starting with 6387f0ba111eeefb9ed1d08a0b03866fbb813a63d9b0efa92a4c8f53dc84a07f not found: ID does not exist" Jul 29 20:53:07.653115 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.653088 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b77f4479d-6nvtd"] Jul 29 20:53:07.655139 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:07.655118 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b77f4479d-6nvtd"] Jul 29 20:53:08.492310 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:08.492273 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b558243-4ba4-44ca-9b78-638a1334e45a" path="/var/lib/kubelet/pods/6b558243-4ba4-44ca-9b78-638a1334e45a/volumes" Jul 29 20:53:08.709977 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:08.709947 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:53:13.724732 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:13.724692 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" podUID="02b94d5a-97fb-4149-ad60-937e3e1f6555" containerName="registry" containerID="cri-o://93f65f7c1abd8d594ef456b3dcefab159bf59ef328e1a10a4f7a41d49595c537" gracePeriod=30 Jul 29 20:53:15.660661 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:15.660627 2582 generic.go:356] "Generic (PLEG): container finished" podID="02b94d5a-97fb-4149-ad60-937e3e1f6555" containerID="93f65f7c1abd8d594ef456b3dcefab159bf59ef328e1a10a4f7a41d49595c537" exitCode=0 Jul 29 20:53:15.661087 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:15.660679 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" event={"ID":"02b94d5a-97fb-4149-ad60-937e3e1f6555","Type":"ContainerDied","Data":"93f65f7c1abd8d594ef456b3dcefab159bf59ef328e1a10a4f7a41d49595c537"} Jul 29 20:53:15.661087 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:15.660716 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" event={"ID":"02b94d5a-97fb-4149-ad60-937e3e1f6555","Type":"ContainerStarted","Data":"9b953cfa8eecde4cb579d05edd9dad3bd4bf06673922667e94d2053f2420c930"} Jul 29 20:53:15.661087 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:15.660787 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:53:36.668908 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:36.668883 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c8c48fd5c-mhvt5" Jul 29 20:53:46.328616 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:46.328579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:53:46.331036 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:46.331004 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b760b0-11c0-4512-aa57-fd441bdfd5a1-metrics-certs\") pod \"network-metrics-daemon-km6rd\" (UID: \"52b760b0-11c0-4512-aa57-fd441bdfd5a1\") " pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:53:46.590042 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:46.589956 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vp87c\"" Jul 29 20:53:46.597977 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:46.597957 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-km6rd" Jul 29 20:53:46.721644 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:46.721616 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-km6rd"] Jul 29 20:53:46.724792 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:53:46.724760 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b760b0_11c0_4512_aa57_fd441bdfd5a1.slice/crio-c3ddf5512888d4e47b2c7bc05e23ed58cddb50d1a0e3e117cfb5c6191e3e8ba0 WatchSource:0}: Error finding container c3ddf5512888d4e47b2c7bc05e23ed58cddb50d1a0e3e117cfb5c6191e3e8ba0: Status 404 returned error can't find the container with id c3ddf5512888d4e47b2c7bc05e23ed58cddb50d1a0e3e117cfb5c6191e3e8ba0 Jul 29 20:53:46.726671 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:46.726654 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 20:53:46.760562 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:46.760539 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-km6rd" event={"ID":"52b760b0-11c0-4512-aa57-fd441bdfd5a1","Type":"ContainerStarted","Data":"c3ddf5512888d4e47b2c7bc05e23ed58cddb50d1a0e3e117cfb5c6191e3e8ba0"} Jul 29 20:53:47.765558 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:47.765474 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-km6rd" event={"ID":"52b760b0-11c0-4512-aa57-fd441bdfd5a1","Type":"ContainerStarted","Data":"08d1e0fc4495b4e5d92e0201b3088c1e241454a550be9228f42b879630a45f77"} Jul 29 20:53:47.765558 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:47.765531 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-km6rd" event={"ID":"52b760b0-11c0-4512-aa57-fd441bdfd5a1","Type":"ContainerStarted","Data":"c20af25565cc393f0ae84978524d6693c23ea76742aed43277a7937e7016bd52"} Jul 29 20:53:47.782901 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:47.782854 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-km6rd" podStartSLOduration=375.006337557 podStartE2EDuration="6m15.782839467s" totalImagesPullingTime="776.50191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:53:47.782286833 +0000 UTC m=+375.905876675" watchObservedRunningTime="2026-07-29 20:53:47.782839467 +0000 UTC m=+375.906429309" Jul 29 20:53:48.556225 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:53:48.556188 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-755d7586fd-nhdhj"] Jul 29 20:54:13.575320 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.575265 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-755d7586fd-nhdhj" podUID="91887504-5ac3-40e4-b834-e4f4fb2e3c20" containerName="console" containerID="cri-o://2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507" gracePeriod=15 Jul 29 20:54:13.817986 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.817958 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:54:13.828010 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.827958 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-service-ca\") pod \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " Jul 29 20:54:13.828010 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.827985 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-serving-cert\") pod \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " Jul 29 20:54:13.828169 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.828046 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-config\") pod \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " Jul 29 20:54:13.828169 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.828072 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg7zp\" (UniqueName: \"kubernetes.io/projected/91887504-5ac3-40e4-b834-e4f4fb2e3c20-kube-api-access-vg7zp\") pod \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " Jul 29 20:54:13.828308 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.828290 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-oauth-serving-cert\") pod \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " Jul 29 20:54:13.828368 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.828333 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-trusted-ca-bundle\") pod \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " Jul 29 20:54:13.828421 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.828365 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-oauth-config\") pod \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\" (UID: \"91887504-5ac3-40e4-b834-e4f4fb2e3c20\") " Jul 29 20:54:13.828421 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.828388 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-service-ca" (OuterVolumeSpecName: "service-ca") pod "91887504-5ac3-40e4-b834-e4f4fb2e3c20" (UID: "91887504-5ac3-40e4-b834-e4f4fb2e3c20"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:54:13.828514 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.828474 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-config" (OuterVolumeSpecName: "console-config") pod "91887504-5ac3-40e4-b834-e4f4fb2e3c20" (UID: "91887504-5ac3-40e4-b834-e4f4fb2e3c20"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:54:13.828681 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.828622 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-config\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:54:13.828681 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.828641 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-service-ca\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:54:13.828681 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.828660 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "91887504-5ac3-40e4-b834-e4f4fb2e3c20" (UID: "91887504-5ac3-40e4-b834-e4f4fb2e3c20"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:54:13.829051 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.828731 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "91887504-5ac3-40e4-b834-e4f4fb2e3c20" (UID: "91887504-5ac3-40e4-b834-e4f4fb2e3c20"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:54:13.830374 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.830350 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "91887504-5ac3-40e4-b834-e4f4fb2e3c20" (UID: "91887504-5ac3-40e4-b834-e4f4fb2e3c20"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:54:13.830451 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.830371 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91887504-5ac3-40e4-b834-e4f4fb2e3c20-kube-api-access-vg7zp" (OuterVolumeSpecName: "kube-api-access-vg7zp") pod "91887504-5ac3-40e4-b834-e4f4fb2e3c20" (UID: "91887504-5ac3-40e4-b834-e4f4fb2e3c20"). InnerVolumeSpecName "kube-api-access-vg7zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:54:13.830491 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.830477 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "91887504-5ac3-40e4-b834-e4f4fb2e3c20" (UID: "91887504-5ac3-40e4-b834-e4f4fb2e3c20"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:54:13.852004 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.851979 2582 generic.go:356] "Generic (PLEG): container finished" podID="91887504-5ac3-40e4-b834-e4f4fb2e3c20" containerID="2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507" exitCode=0 Jul 29 20:54:13.852154 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.852063 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755d7586fd-nhdhj" event={"ID":"91887504-5ac3-40e4-b834-e4f4fb2e3c20","Type":"ContainerDied","Data":"2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507"} Jul 29 20:54:13.852154 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.852086 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755d7586fd-nhdhj" Jul 29 20:54:13.852154 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.852117 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755d7586fd-nhdhj" event={"ID":"91887504-5ac3-40e4-b834-e4f4fb2e3c20","Type":"ContainerDied","Data":"9a9b6ba900f7c48798fc23c6a18fde6a94ffd3f3db76fbf5e88bd10f46fca506"} Jul 29 20:54:13.852154 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.852147 2582 scope.go:117] "RemoveContainer" containerID="2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507" Jul 29 20:54:13.860051 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.860010 2582 scope.go:117] "RemoveContainer" containerID="2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507" Jul 29 20:54:13.860304 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:54:13.860286 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507\": container with ID starting with 2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507 not found: ID does not exist" containerID="2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507" Jul 29 20:54:13.860370 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.860316 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507"} err="failed to get container status \"2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507\": rpc error: code = NotFound desc = could not find container \"2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507\": container with ID starting with 2b6cf70f53d9ce034d02410eea5a41a85d1553ec6260a7f3deb677af1f8e0507 not found: ID does not exist" Jul 29 20:54:13.873112 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.873090 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-755d7586fd-nhdhj"] Jul 29 20:54:13.876653 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.876635 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-755d7586fd-nhdhj"] Jul 29 20:54:13.929411 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.929389 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-oauth-serving-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:54:13.929411 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.929412 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91887504-5ac3-40e4-b834-e4f4fb2e3c20-trusted-ca-bundle\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:54:13.929548 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.929427 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-oauth-config\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:54:13.929548 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.929439 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91887504-5ac3-40e4-b834-e4f4fb2e3c20-console-serving-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:54:13.929548 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:13.929450 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg7zp\" (UniqueName: \"kubernetes.io/projected/91887504-5ac3-40e4-b834-e4f4fb2e3c20-kube-api-access-vg7zp\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:54:14.491038 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:54:14.490996 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91887504-5ac3-40e4-b834-e4f4fb2e3c20" path="/var/lib/kubelet/pods/91887504-5ac3-40e4-b834-e4f4fb2e3c20/volumes" Jul 29 20:55:07.434373 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.434301 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk"] Jul 29 20:55:07.434812 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.434554 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91887504-5ac3-40e4-b834-e4f4fb2e3c20" containerName="console" Jul 29 20:55:07.434812 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.434566 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91887504-5ac3-40e4-b834-e4f4fb2e3c20" containerName="console" Jul 29 20:55:07.434812 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.434579 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b558243-4ba4-44ca-9b78-638a1334e45a" containerName="console" Jul 29 20:55:07.434812 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.434584 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b558243-4ba4-44ca-9b78-638a1334e45a" containerName="console" Jul 29 20:55:07.434812 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.434628 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="91887504-5ac3-40e4-b834-e4f4fb2e3c20" containerName="console" Jul 29 20:55:07.434812 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.434636 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b558243-4ba4-44ca-9b78-638a1334e45a" containerName="console" Jul 29 20:55:07.437624 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.437605 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:07.440649 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.440627 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 20:55:07.440905 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.440887 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cpktn\"" Jul 29 20:55:07.441763 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.441748 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 20:55:07.454384 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.454364 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk"] Jul 29 20:55:07.582432 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.582404 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d9c3357-ab9e-4691-af2f-ce074df58507-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk\" (UID: \"5d9c3357-ab9e-4691-af2f-ce074df58507\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:07.582563 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.582442 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d9c3357-ab9e-4691-af2f-ce074df58507-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk\" (UID: \"5d9c3357-ab9e-4691-af2f-ce074df58507\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:07.582563 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.582474 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c64pf\" (UniqueName: \"kubernetes.io/projected/5d9c3357-ab9e-4691-af2f-ce074df58507-kube-api-access-c64pf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk\" (UID: \"5d9c3357-ab9e-4691-af2f-ce074df58507\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:07.683197 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.683170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d9c3357-ab9e-4691-af2f-ce074df58507-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk\" (UID: \"5d9c3357-ab9e-4691-af2f-ce074df58507\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:07.683332 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.683215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d9c3357-ab9e-4691-af2f-ce074df58507-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk\" (UID: \"5d9c3357-ab9e-4691-af2f-ce074df58507\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:07.683332 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.683247 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c64pf\" (UniqueName: \"kubernetes.io/projected/5d9c3357-ab9e-4691-af2f-ce074df58507-kube-api-access-c64pf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk\" (UID: \"5d9c3357-ab9e-4691-af2f-ce074df58507\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:07.683539 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.683523 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d9c3357-ab9e-4691-af2f-ce074df58507-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk\" (UID: \"5d9c3357-ab9e-4691-af2f-ce074df58507\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:07.683625 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.683603 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d9c3357-ab9e-4691-af2f-ce074df58507-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk\" (UID: \"5d9c3357-ab9e-4691-af2f-ce074df58507\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:07.692166 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.692124 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c64pf\" (UniqueName: \"kubernetes.io/projected/5d9c3357-ab9e-4691-af2f-ce074df58507-kube-api-access-c64pf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk\" (UID: \"5d9c3357-ab9e-4691-af2f-ce074df58507\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:07.746133 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.746101 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:07.877210 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:07.877182 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk"] Jul 29 20:55:07.879849 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:55:07.879824 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9c3357_ab9e_4691_af2f_ce074df58507.slice/crio-e48758a3420b09c39c230a9ca8420ce5df01b3aa1132c73c4503467168414111 WatchSource:0}: Error finding container e48758a3420b09c39c230a9ca8420ce5df01b3aa1132c73c4503467168414111: Status 404 returned error can't find the container with id e48758a3420b09c39c230a9ca8420ce5df01b3aa1132c73c4503467168414111 Jul 29 20:55:08.031473 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:08.031445 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" event={"ID":"5d9c3357-ab9e-4691-af2f-ce074df58507","Type":"ContainerStarted","Data":"e48758a3420b09c39c230a9ca8420ce5df01b3aa1132c73c4503467168414111"} Jul 29 20:55:14.057195 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:14.057160 2582 generic.go:356] "Generic (PLEG): container finished" podID="5d9c3357-ab9e-4691-af2f-ce074df58507" containerID="e950e1ae7cbfdf51ef81ae4bccf84b54aafc401773ff51bd4ea6128d43470a8f" exitCode=0 Jul 29 20:55:14.057655 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:14.057240 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" event={"ID":"5d9c3357-ab9e-4691-af2f-ce074df58507","Type":"ContainerDied","Data":"e950e1ae7cbfdf51ef81ae4bccf84b54aafc401773ff51bd4ea6128d43470a8f"} Jul 29 20:55:18.073793 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:18.073753 2582 generic.go:356] "Generic (PLEG): container finished" podID="5d9c3357-ab9e-4691-af2f-ce074df58507" containerID="a6a4e488be10561807e9eaef549ce45b59562a15a12a42b50e863486c5b4c915" exitCode=0 Jul 29 20:55:18.074210 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:18.073838 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" event={"ID":"5d9c3357-ab9e-4691-af2f-ce074df58507","Type":"ContainerDied","Data":"a6a4e488be10561807e9eaef549ce45b59562a15a12a42b50e863486c5b4c915"} Jul 29 20:55:20.271008 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.270971 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xqkqh"] Jul 29 20:55:20.287048 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.286999 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xqkqh"] Jul 29 20:55:20.287202 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.287129 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xqkqh" Jul 29 20:55:20.289898 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.289874 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 29 20:55:20.290908 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.290890 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 29 20:55:20.291052 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.290887 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xttb5\"" Jul 29 20:55:20.383503 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.383471 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxx6q\" (UniqueName: \"kubernetes.io/projected/7f682dba-32ce-47cc-8d95-b4c7706945ed-kube-api-access-xxx6q\") pod \"cert-manager-55b4d954d5-xqkqh\" (UID: \"7f682dba-32ce-47cc-8d95-b4c7706945ed\") " pod="cert-manager/cert-manager-55b4d954d5-xqkqh" Jul 29 20:55:20.383686 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.383591 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f682dba-32ce-47cc-8d95-b4c7706945ed-bound-sa-token\") pod \"cert-manager-55b4d954d5-xqkqh\" (UID: \"7f682dba-32ce-47cc-8d95-b4c7706945ed\") " pod="cert-manager/cert-manager-55b4d954d5-xqkqh" Jul 29 20:55:20.484958 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.484920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f682dba-32ce-47cc-8d95-b4c7706945ed-bound-sa-token\") pod \"cert-manager-55b4d954d5-xqkqh\" (UID: \"7f682dba-32ce-47cc-8d95-b4c7706945ed\") " pod="cert-manager/cert-manager-55b4d954d5-xqkqh" Jul 29 20:55:20.485170 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.484974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxx6q\" (UniqueName: \"kubernetes.io/projected/7f682dba-32ce-47cc-8d95-b4c7706945ed-kube-api-access-xxx6q\") pod \"cert-manager-55b4d954d5-xqkqh\" (UID: \"7f682dba-32ce-47cc-8d95-b4c7706945ed\") " pod="cert-manager/cert-manager-55b4d954d5-xqkqh" Jul 29 20:55:20.493535 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.493503 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f682dba-32ce-47cc-8d95-b4c7706945ed-bound-sa-token\") pod \"cert-manager-55b4d954d5-xqkqh\" (UID: \"7f682dba-32ce-47cc-8d95-b4c7706945ed\") " pod="cert-manager/cert-manager-55b4d954d5-xqkqh" Jul 29 20:55:20.494008 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.493989 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxx6q\" (UniqueName: \"kubernetes.io/projected/7f682dba-32ce-47cc-8d95-b4c7706945ed-kube-api-access-xxx6q\") pod \"cert-manager-55b4d954d5-xqkqh\" (UID: \"7f682dba-32ce-47cc-8d95-b4c7706945ed\") " pod="cert-manager/cert-manager-55b4d954d5-xqkqh" Jul 29 20:55:20.599451 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.599203 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xqkqh" Jul 29 20:55:20.746498 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:20.746446 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xqkqh"] Jul 29 20:55:20.749130 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:55:20.749103 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f682dba_32ce_47cc_8d95_b4c7706945ed.slice/crio-29dfba007a7202da529a8a25460b1a024f1620ccea885c52f4dffb14019e9572 WatchSource:0}: Error finding container 29dfba007a7202da529a8a25460b1a024f1620ccea885c52f4dffb14019e9572: Status 404 returned error can't find the container with id 29dfba007a7202da529a8a25460b1a024f1620ccea885c52f4dffb14019e9572 Jul 29 20:55:21.086159 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:21.086122 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xqkqh" event={"ID":"7f682dba-32ce-47cc-8d95-b4c7706945ed","Type":"ContainerStarted","Data":"29dfba007a7202da529a8a25460b1a024f1620ccea885c52f4dffb14019e9572"} Jul 29 20:55:27.110286 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:27.110245 2582 generic.go:356] "Generic (PLEG): container finished" podID="5d9c3357-ab9e-4691-af2f-ce074df58507" containerID="202616f2da78d966b2943bee87410548f9992364658c5ca1f662238ba4594786" exitCode=0 Jul 29 20:55:27.110728 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:27.110331 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" event={"ID":"5d9c3357-ab9e-4691-af2f-ce074df58507","Type":"ContainerDied","Data":"202616f2da78d966b2943bee87410548f9992364658c5ca1f662238ba4594786"} Jul 29 20:55:27.111585 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:27.111565 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xqkqh" event={"ID":"7f682dba-32ce-47cc-8d95-b4c7706945ed","Type":"ContainerStarted","Data":"5402a92c596d99b8a932bdb397c9404452f1f8f7a03baba55b5029bad7f21d1e"} Jul 29 20:55:27.142550 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:27.142506 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xqkqh" podStartSLOduration=1.792508695 podStartE2EDuration="7.142486015s" totalImagesPullingTime="5.34997732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:55:27.142094852 +0000 UTC m=+475.265684694" watchObservedRunningTime="2026-07-29 20:55:27.142486015 +0000 UTC m=+475.266075857" Jul 29 20:55:28.235380 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:28.235361 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:28.352332 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:28.352301 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c64pf\" (UniqueName: \"kubernetes.io/projected/5d9c3357-ab9e-4691-af2f-ce074df58507-kube-api-access-c64pf\") pod \"5d9c3357-ab9e-4691-af2f-ce074df58507\" (UID: \"5d9c3357-ab9e-4691-af2f-ce074df58507\") " Jul 29 20:55:28.352488 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:28.352341 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d9c3357-ab9e-4691-af2f-ce074df58507-bundle\") pod \"5d9c3357-ab9e-4691-af2f-ce074df58507\" (UID: \"5d9c3357-ab9e-4691-af2f-ce074df58507\") " Jul 29 20:55:28.352488 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:28.352392 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d9c3357-ab9e-4691-af2f-ce074df58507-util\") pod \"5d9c3357-ab9e-4691-af2f-ce074df58507\" (UID: \"5d9c3357-ab9e-4691-af2f-ce074df58507\") " Jul 29 20:55:28.352708 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:28.352680 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9c3357-ab9e-4691-af2f-ce074df58507-bundle" (OuterVolumeSpecName: "bundle") pod "5d9c3357-ab9e-4691-af2f-ce074df58507" (UID: "5d9c3357-ab9e-4691-af2f-ce074df58507"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:55:28.354630 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:28.354613 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9c3357-ab9e-4691-af2f-ce074df58507-kube-api-access-c64pf" (OuterVolumeSpecName: "kube-api-access-c64pf") pod "5d9c3357-ab9e-4691-af2f-ce074df58507" (UID: "5d9c3357-ab9e-4691-af2f-ce074df58507"). InnerVolumeSpecName "kube-api-access-c64pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:55:28.356580 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:28.356556 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9c3357-ab9e-4691-af2f-ce074df58507-util" (OuterVolumeSpecName: "util") pod "5d9c3357-ab9e-4691-af2f-ce074df58507" (UID: "5d9c3357-ab9e-4691-af2f-ce074df58507"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:55:28.453750 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:28.453676 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d9c3357-ab9e-4691-af2f-ce074df58507-util\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:55:28.453750 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:28.453703 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c64pf\" (UniqueName: \"kubernetes.io/projected/5d9c3357-ab9e-4691-af2f-ce074df58507-kube-api-access-c64pf\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:55:28.453750 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:28.453715 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d9c3357-ab9e-4691-af2f-ce074df58507-bundle\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:55:29.121375 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:29.121337 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" event={"ID":"5d9c3357-ab9e-4691-af2f-ce074df58507","Type":"ContainerDied","Data":"e48758a3420b09c39c230a9ca8420ce5df01b3aa1132c73c4503467168414111"} Jul 29 20:55:29.121375 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:29.121354 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4qnk" Jul 29 20:55:29.121577 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:29.121405 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e48758a3420b09c39c230a9ca8420ce5df01b3aa1132c73c4503467168414111" Jul 29 20:55:55.489312 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.489283 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp"] Jul 29 20:55:55.489719 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.489504 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d9c3357-ab9e-4691-af2f-ce074df58507" containerName="util" Jul 29 20:55:55.489719 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.489514 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d9c3357-ab9e-4691-af2f-ce074df58507" containerName="util" Jul 29 20:55:55.489719 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.489524 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d9c3357-ab9e-4691-af2f-ce074df58507" containerName="pull" Jul 29 20:55:55.489719 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.489528 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d9c3357-ab9e-4691-af2f-ce074df58507" containerName="pull" Jul 29 20:55:55.489719 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.489538 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d9c3357-ab9e-4691-af2f-ce074df58507" containerName="extract" Jul 29 20:55:55.489719 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.489543 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d9c3357-ab9e-4691-af2f-ce074df58507" containerName="extract" Jul 29 20:55:55.489719 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.489585 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d9c3357-ab9e-4691-af2f-ce074df58507" containerName="extract" Jul 29 20:55:55.510458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.510432 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.512201 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.512175 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp"] Jul 29 20:55:55.513225 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.513202 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 20:55:55.513653 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.513632 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 20:55:55.513751 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.513677 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 20:55:55.513967 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.513951 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 20:55:55.514075 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.513987 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-f44g6\"" Jul 29 20:55:55.528779 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.528760 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmf7\" (UniqueName: \"kubernetes.io/projected/9e33cf81-e1d6-442f-a673-8a032a0b0b14-kube-api-access-dhmf7\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.528865 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.528788 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e33cf81-e1d6-442f-a673-8a032a0b0b14-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.528865 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.528816 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e33cf81-e1d6-442f-a673-8a032a0b0b14-tmp\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.528933 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.528866 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e33cf81-e1d6-442f-a673-8a032a0b0b14-webhook-cert\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.528933 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.528905 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e33cf81-e1d6-442f-a673-8a032a0b0b14-manifests\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.630078 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.630019 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e33cf81-e1d6-442f-a673-8a032a0b0b14-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.630252 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.630110 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e33cf81-e1d6-442f-a673-8a032a0b0b14-tmp\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.630252 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.630135 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e33cf81-e1d6-442f-a673-8a032a0b0b14-webhook-cert\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.630252 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.630166 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e33cf81-e1d6-442f-a673-8a032a0b0b14-manifests\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.630409 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.630283 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmf7\" (UniqueName: \"kubernetes.io/projected/9e33cf81-e1d6-442f-a673-8a032a0b0b14-kube-api-access-dhmf7\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.630602 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.630571 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e33cf81-e1d6-442f-a673-8a032a0b0b14-manifests\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.632550 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.632526 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e33cf81-e1d6-442f-a673-8a032a0b0b14-tmp\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.632616 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.632586 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e33cf81-e1d6-442f-a673-8a032a0b0b14-webhook-cert\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.632660 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.632620 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e33cf81-e1d6-442f-a673-8a032a0b0b14-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.639547 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.639529 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmf7\" (UniqueName: \"kubernetes.io/projected/9e33cf81-e1d6-442f-a673-8a032a0b0b14-kube-api-access-dhmf7\") pod \"opendatahub-operator-controller-manager-57645d8769-6tbhp\" (UID: \"9e33cf81-e1d6-442f-a673-8a032a0b0b14\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.821446 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.821374 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:55:55.982748 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:55.982725 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp"] Jul 29 20:55:55.985524 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:55:55.985496 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e33cf81_e1d6_442f_a673_8a032a0b0b14.slice/crio-60f03638a59292af84de19237dc4927fe5ac6ae0e31269ddea9d29efb8c4fbc5 WatchSource:0}: Error finding container 60f03638a59292af84de19237dc4927fe5ac6ae0e31269ddea9d29efb8c4fbc5: Status 404 returned error can't find the container with id 60f03638a59292af84de19237dc4927fe5ac6ae0e31269ddea9d29efb8c4fbc5 Jul 29 20:55:56.215275 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:56.215200 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" event={"ID":"9e33cf81-e1d6-442f-a673-8a032a0b0b14","Type":"ContainerStarted","Data":"60f03638a59292af84de19237dc4927fe5ac6ae0e31269ddea9d29efb8c4fbc5"} Jul 29 20:55:59.228285 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:55:59.228252 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" event={"ID":"9e33cf81-e1d6-442f-a673-8a032a0b0b14","Type":"ContainerStarted","Data":"08a75b6b693442163398f128e0a227a1c9e900e6bfc58beb0ef98e99c3cc221e"} Jul 29 20:56:01.237147 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:01.237118 2582 generic.go:356] "Generic (PLEG): container finished" podID="9e33cf81-e1d6-442f-a673-8a032a0b0b14" containerID="08a75b6b693442163398f128e0a227a1c9e900e6bfc58beb0ef98e99c3cc221e" exitCode=0 Jul 29 20:56:01.237530 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:01.237188 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" event={"ID":"9e33cf81-e1d6-442f-a673-8a032a0b0b14","Type":"ContainerDied","Data":"08a75b6b693442163398f128e0a227a1c9e900e6bfc58beb0ef98e99c3cc221e"} Jul 29 20:56:02.242270 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:02.242235 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" event={"ID":"9e33cf81-e1d6-442f-a673-8a032a0b0b14","Type":"ContainerStarted","Data":"b904c52045fe3c115da95a0e6253a336a450e3d69e0027cc527c77aa239dcc46"} Jul 29 20:56:02.242684 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:02.242534 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:56:02.267564 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:02.267518 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" podStartSLOduration=2.666775441 podStartE2EDuration="7.267504149s" totalImagesPullingTime="3.517765138s" totalInitContainerRuntime="1.08296357s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:55:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:02.26531275 +0000 UTC m=+510.388902592" watchObservedRunningTime="2026-07-29 20:56:02.267504149 +0000 UTC m=+510.391093991" Jul 29 20:56:13.248079 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:13.248051 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-6tbhp" Jul 29 20:56:15.396885 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.396855 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x"] Jul 29 20:56:15.402494 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.402479 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" Jul 29 20:56:15.404732 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.404701 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 20:56:15.405082 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.405068 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 20:56:15.405726 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.405710 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 20:56:15.407056 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.407038 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 20:56:15.416924 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.416900 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x"] Jul 29 20:56:15.474365 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.474338 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69af1ab8-f326-4137-9b3c-9cb8638ccf25-tls-certs\") pod \"kube-auth-proxy-68db9f4dd4-lrl5x\" (UID: \"69af1ab8-f326-4137-9b3c-9cb8638ccf25\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" Jul 29 20:56:15.474481 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.474405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69af1ab8-f326-4137-9b3c-9cb8638ccf25-tmp\") pod \"kube-auth-proxy-68db9f4dd4-lrl5x\" (UID: \"69af1ab8-f326-4137-9b3c-9cb8638ccf25\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" Jul 29 20:56:15.474481 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.474464 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cqbz\" (UniqueName: \"kubernetes.io/projected/69af1ab8-f326-4137-9b3c-9cb8638ccf25-kube-api-access-5cqbz\") pod \"kube-auth-proxy-68db9f4dd4-lrl5x\" (UID: \"69af1ab8-f326-4137-9b3c-9cb8638ccf25\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" Jul 29 20:56:15.567327 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.567299 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8r9ws"] Jul 29 20:56:15.569489 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.569473 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" Jul 29 20:56:15.571982 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.571960 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 29 20:56:15.572103 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.571962 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wl8zs\"" Jul 29 20:56:15.575362 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.575343 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cqbz\" (UniqueName: \"kubernetes.io/projected/69af1ab8-f326-4137-9b3c-9cb8638ccf25-kube-api-access-5cqbz\") pod \"kube-auth-proxy-68db9f4dd4-lrl5x\" (UID: \"69af1ab8-f326-4137-9b3c-9cb8638ccf25\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" Jul 29 20:56:15.575502 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.575382 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69af1ab8-f326-4137-9b3c-9cb8638ccf25-tls-certs\") pod \"kube-auth-proxy-68db9f4dd4-lrl5x\" (UID: \"69af1ab8-f326-4137-9b3c-9cb8638ccf25\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" Jul 29 20:56:15.575502 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.575446 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69af1ab8-f326-4137-9b3c-9cb8638ccf25-tmp\") pod \"kube-auth-proxy-68db9f4dd4-lrl5x\" (UID: \"69af1ab8-f326-4137-9b3c-9cb8638ccf25\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" Jul 29 20:56:15.575502 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:56:15.575477 2582 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 29 20:56:15.575730 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:56:15.575536 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69af1ab8-f326-4137-9b3c-9cb8638ccf25-tls-certs podName:69af1ab8-f326-4137-9b3c-9cb8638ccf25 nodeName:}" failed. No retries permitted until 2026-07-29 20:56:16.075518357 +0000 UTC m=+524.199108183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/69af1ab8-f326-4137-9b3c-9cb8638ccf25-tls-certs") pod "kube-auth-proxy-68db9f4dd4-lrl5x" (UID: "69af1ab8-f326-4137-9b3c-9cb8638ccf25") : secret "kube-auth-proxy-tls" not found Jul 29 20:56:15.578125 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.578106 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69af1ab8-f326-4137-9b3c-9cb8638ccf25-tmp\") pod \"kube-auth-proxy-68db9f4dd4-lrl5x\" (UID: \"69af1ab8-f326-4137-9b3c-9cb8638ccf25\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" Jul 29 20:56:15.582541 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.582521 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8r9ws"] Jul 29 20:56:15.592558 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.592533 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cqbz\" (UniqueName: \"kubernetes.io/projected/69af1ab8-f326-4137-9b3c-9cb8638ccf25-kube-api-access-5cqbz\") pod \"kube-auth-proxy-68db9f4dd4-lrl5x\" (UID: \"69af1ab8-f326-4137-9b3c-9cb8638ccf25\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" Jul 29 20:56:15.676315 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.676247 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf4sv\" (UniqueName: \"kubernetes.io/projected/28c6a372-aafb-4449-84b9-81a3263ab62d-kube-api-access-rf4sv\") pod \"odh-model-controller-7998c67dc4-8r9ws\" (UID: \"28c6a372-aafb-4449-84b9-81a3263ab62d\") " pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" Jul 29 20:56:15.676315 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.676288 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28c6a372-aafb-4449-84b9-81a3263ab62d-cert\") pod \"odh-model-controller-7998c67dc4-8r9ws\" (UID: \"28c6a372-aafb-4449-84b9-81a3263ab62d\") " pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" Jul 29 20:56:15.776920 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.776897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf4sv\" (UniqueName: \"kubernetes.io/projected/28c6a372-aafb-4449-84b9-81a3263ab62d-kube-api-access-rf4sv\") pod \"odh-model-controller-7998c67dc4-8r9ws\" (UID: \"28c6a372-aafb-4449-84b9-81a3263ab62d\") " pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" Jul 29 20:56:15.777090 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.776931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28c6a372-aafb-4449-84b9-81a3263ab62d-cert\") pod \"odh-model-controller-7998c67dc4-8r9ws\" (UID: \"28c6a372-aafb-4449-84b9-81a3263ab62d\") " pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" Jul 29 20:56:15.779426 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.779404 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28c6a372-aafb-4449-84b9-81a3263ab62d-cert\") pod \"odh-model-controller-7998c67dc4-8r9ws\" (UID: \"28c6a372-aafb-4449-84b9-81a3263ab62d\") " pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" Jul 29 20:56:15.787156 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.787137 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf4sv\" (UniqueName: \"kubernetes.io/projected/28c6a372-aafb-4449-84b9-81a3263ab62d-kube-api-access-rf4sv\") pod \"odh-model-controller-7998c67dc4-8r9ws\" (UID: \"28c6a372-aafb-4449-84b9-81a3263ab62d\") " pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" Jul 29 20:56:15.880948 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:15.880921 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" Jul 29 20:56:16.004830 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.004807 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8r9ws"] Jul 29 20:56:16.007371 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:56:16.007347 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c6a372_aafb_4449_84b9_81a3263ab62d.slice/crio-a23a40c16623467a68c97bbd32f5a9b83a4a96b9daee0e4897568c6a9bff3b9f WatchSource:0}: Error finding container a23a40c16623467a68c97bbd32f5a9b83a4a96b9daee0e4897568c6a9bff3b9f: Status 404 returned error can't find the container with id a23a40c16623467a68c97bbd32f5a9b83a4a96b9daee0e4897568c6a9bff3b9f Jul 29 20:56:16.079336 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.079306 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69af1ab8-f326-4137-9b3c-9cb8638ccf25-tls-certs\") pod \"kube-auth-proxy-68db9f4dd4-lrl5x\" (UID: \"69af1ab8-f326-4137-9b3c-9cb8638ccf25\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" Jul 29 20:56:16.081662 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.081635 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69af1ab8-f326-4137-9b3c-9cb8638ccf25-tls-certs\") pod \"kube-auth-proxy-68db9f4dd4-lrl5x\" (UID: \"69af1ab8-f326-4137-9b3c-9cb8638ccf25\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" Jul 29 20:56:16.177152 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.177119 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nn9t2"] Jul 29 20:56:16.180004 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.179988 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" Jul 29 20:56:16.185267 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.185243 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 29 20:56:16.186193 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.186171 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rmjl8\"" Jul 29 20:56:16.191141 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.191119 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nn9t2"] Jul 29 20:56:16.281388 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.281354 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9dfe876-9ede-47bb-87f6-d9d9fd916035-tls-certs\") pod \"model-serving-api-54d89bd5fc-nn9t2\" (UID: \"e9dfe876-9ede-47bb-87f6-d9d9fd916035\") " pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" Jul 29 20:56:16.281539 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.281406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkhgh\" (UniqueName: \"kubernetes.io/projected/e9dfe876-9ede-47bb-87f6-d9d9fd916035-kube-api-access-fkhgh\") pod \"model-serving-api-54d89bd5fc-nn9t2\" (UID: \"e9dfe876-9ede-47bb-87f6-d9d9fd916035\") " pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" Jul 29 20:56:16.293271 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.293242 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" event={"ID":"28c6a372-aafb-4449-84b9-81a3263ab62d","Type":"ContainerStarted","Data":"a23a40c16623467a68c97bbd32f5a9b83a4a96b9daee0e4897568c6a9bff3b9f"} Jul 29 20:56:16.311356 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.311328 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" Jul 29 20:56:16.382734 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.382709 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9dfe876-9ede-47bb-87f6-d9d9fd916035-tls-certs\") pod \"model-serving-api-54d89bd5fc-nn9t2\" (UID: \"e9dfe876-9ede-47bb-87f6-d9d9fd916035\") " pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" Jul 29 20:56:16.382886 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.382756 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkhgh\" (UniqueName: \"kubernetes.io/projected/e9dfe876-9ede-47bb-87f6-d9d9fd916035-kube-api-access-fkhgh\") pod \"model-serving-api-54d89bd5fc-nn9t2\" (UID: \"e9dfe876-9ede-47bb-87f6-d9d9fd916035\") " pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" Jul 29 20:56:16.382886 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:56:16.382858 2582 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 29 20:56:16.382965 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:56:16.382930 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9dfe876-9ede-47bb-87f6-d9d9fd916035-tls-certs podName:e9dfe876-9ede-47bb-87f6-d9d9fd916035 nodeName:}" failed. No retries permitted until 2026-07-29 20:56:16.882911904 +0000 UTC m=+525.006501725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e9dfe876-9ede-47bb-87f6-d9d9fd916035-tls-certs") pod "model-serving-api-54d89bd5fc-nn9t2" (UID: "e9dfe876-9ede-47bb-87f6-d9d9fd916035") : secret "model-serving-api-tls" not found Jul 29 20:56:16.397949 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.397916 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkhgh\" (UniqueName: \"kubernetes.io/projected/e9dfe876-9ede-47bb-87f6-d9d9fd916035-kube-api-access-fkhgh\") pod \"model-serving-api-54d89bd5fc-nn9t2\" (UID: \"e9dfe876-9ede-47bb-87f6-d9d9fd916035\") " pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" Jul 29 20:56:16.437819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.437790 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x"] Jul 29 20:56:16.439962 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:56:16.439935 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69af1ab8_f326_4137_9b3c_9cb8638ccf25.slice/crio-3bcfe8b5d447df607dd43ad90b592f1493a1adb60d4e809f6c1049398798fdc0 WatchSource:0}: Error finding container 3bcfe8b5d447df607dd43ad90b592f1493a1adb60d4e809f6c1049398798fdc0: Status 404 returned error can't find the container with id 3bcfe8b5d447df607dd43ad90b592f1493a1adb60d4e809f6c1049398798fdc0 Jul 29 20:56:16.868307 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.868276 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx"] Jul 29 20:56:16.870710 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.870690 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:16.873226 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.873206 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 20:56:16.873226 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.873223 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rtc4z\"" Jul 29 20:56:16.873392 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.873261 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 20:56:16.873392 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.873206 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 20:56:16.873392 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.873206 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 20:56:16.887524 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.887496 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9dfe876-9ede-47bb-87f6-d9d9fd916035-tls-certs\") pod \"model-serving-api-54d89bd5fc-nn9t2\" (UID: \"e9dfe876-9ede-47bb-87f6-d9d9fd916035\") " pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" Jul 29 20:56:16.890797 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.890774 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9dfe876-9ede-47bb-87f6-d9d9fd916035-tls-certs\") pod \"model-serving-api-54d89bd5fc-nn9t2\" (UID: \"e9dfe876-9ede-47bb-87f6-d9d9fd916035\") " pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" Jul 29 20:56:16.891008 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.890976 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx"] Jul 29 20:56:16.988603 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.988571 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:16.988778 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.988622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:16.988778 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.988641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:16.988778 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.988658 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxv2z\" (UniqueName: \"kubernetes.io/projected/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-kube-api-access-dxv2z\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:16.988964 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.988791 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:16.988964 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.988841 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:16.988964 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:16.988878 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.089726 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.089688 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" Jul 29 20:56:17.089916 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.089853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.089916 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.089892 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.090056 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.089915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxv2z\" (UniqueName: \"kubernetes.io/projected/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-kube-api-access-dxv2z\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.090056 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.089992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.090594 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.090441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.090594 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.090489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.090594 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.090530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.091370 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.090983 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.093419 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.093376 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.095800 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.095781 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.096534 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.096486 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.097972 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.097930 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.103358 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.103311 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxv2z\" (UniqueName: \"kubernetes.io/projected/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-kube-api-access-dxv2z\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.103711 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.103671 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-n4wzx\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.182054 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.181762 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:17.278806 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.278691 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nn9t2"] Jul 29 20:56:17.302283 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.302181 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" event={"ID":"69af1ab8-f326-4137-9b3c-9cb8638ccf25","Type":"ContainerStarted","Data":"3bcfe8b5d447df607dd43ad90b592f1493a1adb60d4e809f6c1049398798fdc0"} Jul 29 20:56:17.308901 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.308829 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" event={"ID":"e9dfe876-9ede-47bb-87f6-d9d9fd916035","Type":"ContainerStarted","Data":"097b87e82084a944a64a886b3f7b47bfd37d2beb9469ea274b71da4c2bffecf6"} Jul 29 20:56:17.379323 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:17.379242 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx"] Jul 29 20:56:17.383290 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:56:17.383255 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533d8e63_4a3d_4c8c_a3ed_fba6cb368623.slice/crio-02624a16579ac9457f7b4a3ba85b0b4293867f0c56ceca21ee9590805d8ffe87 WatchSource:0}: Error finding container 02624a16579ac9457f7b4a3ba85b0b4293867f0c56ceca21ee9590805d8ffe87: Status 404 returned error can't find the container with id 02624a16579ac9457f7b4a3ba85b0b4293867f0c56ceca21ee9590805d8ffe87 Jul 29 20:56:18.314936 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:18.314894 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" event={"ID":"533d8e63-4a3d-4c8c-a3ed-fba6cb368623","Type":"ContainerStarted","Data":"02624a16579ac9457f7b4a3ba85b0b4293867f0c56ceca21ee9590805d8ffe87"} Jul 29 20:56:21.243578 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.243532 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 20:56:21.243948 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.243630 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 20:56:21.332815 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.332776 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" event={"ID":"69af1ab8-f326-4137-9b3c-9cb8638ccf25","Type":"ContainerStarted","Data":"8ae28c979e18d31be7e324527b5f46e4d06a66ba01df67ca3d419de36247d939"} Jul 29 20:56:21.335477 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.335447 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" event={"ID":"533d8e63-4a3d-4c8c-a3ed-fba6cb368623","Type":"ContainerStarted","Data":"964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26"} Jul 29 20:56:21.335604 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.335513 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:21.337403 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.337373 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" event={"ID":"e9dfe876-9ede-47bb-87f6-d9d9fd916035","Type":"ContainerStarted","Data":"e6c14c0471b8c381cbbaa4281df6ee3b3099fdf104cde48760e85f386e99d989"} Jul 29 20:56:21.337538 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.337465 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" Jul 29 20:56:21.338937 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.338915 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" event={"ID":"28c6a372-aafb-4449-84b9-81a3263ab62d","Type":"ContainerStarted","Data":"07654dff38f82893521aeed3d12dc85b35539f87feadf2a51ddb23ad4d2c4c5d"} Jul 29 20:56:21.339100 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.339086 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" Jul 29 20:56:21.350460 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.350415 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-lrl5x" podStartSLOduration=1.592068119 podStartE2EDuration="6.350400262s" totalImagesPullingTime="4.758332143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:21.349429853 +0000 UTC m=+529.473019695" watchObservedRunningTime="2026-07-29 20:56:21.350400262 +0000 UTC m=+529.473990104" Jul 29 20:56:21.365404 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.365352 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" podStartSLOduration=1.447956241 podStartE2EDuration="5.36533559s" totalImagesPullingTime="3.917379349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:21.364249547 +0000 UTC m=+529.487839389" watchObservedRunningTime="2026-07-29 20:56:21.36533559 +0000 UTC m=+529.488925432" Jul 29 20:56:21.382002 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.381943 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" podStartSLOduration=1.188444998 podStartE2EDuration="6.381926191s" totalImagesPullingTime="5.193481193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:21.381095908 +0000 UTC m=+529.504685750" watchObservedRunningTime="2026-07-29 20:56:21.381926191 +0000 UTC m=+529.505516064" Jul 29 20:56:21.405604 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:21.402439 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" podStartSLOduration=1.545282467 podStartE2EDuration="5.402416329s" totalImagesPullingTime="3.857133862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:21.398495467 +0000 UTC m=+529.522085308" watchObservedRunningTime="2026-07-29 20:56:21.402416329 +0000 UTC m=+529.526006172" Jul 29 20:56:22.348278 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:22.348247 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-n4wzx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 20:56:22.348706 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:22.348305 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" podUID="533d8e63-4a3d-4c8c-a3ed-fba6cb368623" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 20:56:24.067654 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.067570 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk"] Jul 29 20:56:24.069594 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.069578 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.072878 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.072855 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lq2zk\"" Jul 29 20:56:24.102527 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.102503 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk"] Jul 29 20:56:24.154334 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.154300 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.154334 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.154336 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.154542 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.154360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.154542 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.154380 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.154542 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.154493 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.154542 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.154522 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.154542 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.154538 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trhb\" (UniqueName: \"kubernetes.io/projected/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-kube-api-access-9trhb\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.154731 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.154591 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.154731 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.154612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256095 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256005 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256290 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256290 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256290 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256153 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trhb\" (UniqueName: \"kubernetes.io/projected/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-kube-api-access-9trhb\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256465 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256465 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256465 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256399 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256465 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256435 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256673 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256468 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256673 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256495 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256774 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256753 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256874 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256853 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.256938 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.256874 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.257367 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.257344 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.258789 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.258765 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.259181 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.259159 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.271332 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.271307 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trhb\" (UniqueName: \"kubernetes.io/projected/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-kube-api-access-9trhb\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.274336 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.274317 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk\" (UID: \"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.379263 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.379148 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:24.523214 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:24.523149 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk"] Jul 29 20:56:24.524949 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:56:24.524926 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f86cc40_8ad9_4593_b2aa_4b72f7d6a8f0.slice/crio-8b7d8f1bcc95da11c529d615f2d75526f148698064f7c2c2ade8f98d826f914e WatchSource:0}: Error finding container 8b7d8f1bcc95da11c529d615f2d75526f148698064f7c2c2ade8f98d826f914e: Status 404 returned error can't find the container with id 8b7d8f1bcc95da11c529d615f2d75526f148698064f7c2c2ade8f98d826f914e Jul 29 20:56:25.347882 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.347847 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:25.359820 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.359794 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" event={"ID":"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0","Type":"ContainerStarted","Data":"8b7d8f1bcc95da11c529d615f2d75526f148698064f7c2c2ade8f98d826f914e"} Jul 29 20:56:25.803410 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.803370 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-twx5p"] Jul 29 20:56:25.807143 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.807121 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" Jul 29 20:56:25.811120 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.811105 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 29 20:56:25.811370 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.811355 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-86t2t\"" Jul 29 20:56:25.822115 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.822089 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-twx5p"] Jul 29 20:56:25.823696 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.823677 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l"] Jul 29 20:56:25.831047 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.828291 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" Jul 29 20:56:25.831754 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.831729 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 29 20:56:25.831904 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.831888 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-xgfg4\"" Jul 29 20:56:25.839992 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.839966 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l"] Jul 29 20:56:25.867915 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.867891 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9khfr\" (UniqueName: \"kubernetes.io/projected/9dae9081-9a2d-4867-b0ca-8697d8fe9b77-kube-api-access-9khfr\") pod \"kserve-controller-manager-d6f4b65b8-twx5p\" (UID: \"9dae9081-9a2d-4867-b0ca-8697d8fe9b77\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" Jul 29 20:56:25.868055 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.867957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dae9081-9a2d-4867-b0ca-8697d8fe9b77-cert\") pod \"kserve-controller-manager-d6f4b65b8-twx5p\" (UID: \"9dae9081-9a2d-4867-b0ca-8697d8fe9b77\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" Jul 29 20:56:25.968937 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.968898 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa1408e5-0576-4fda-aa6b-830280b670f3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-62m6l\" (UID: \"fa1408e5-0576-4fda-aa6b-830280b670f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" Jul 29 20:56:25.969136 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.968946 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9khfr\" (UniqueName: \"kubernetes.io/projected/9dae9081-9a2d-4867-b0ca-8697d8fe9b77-kube-api-access-9khfr\") pod \"kserve-controller-manager-d6f4b65b8-twx5p\" (UID: \"9dae9081-9a2d-4867-b0ca-8697d8fe9b77\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" Jul 29 20:56:25.969136 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.969001 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvfql\" (UniqueName: \"kubernetes.io/projected/fa1408e5-0576-4fda-aa6b-830280b670f3-kube-api-access-hvfql\") pod \"llmisvc-controller-manager-777c6bd7d5-62m6l\" (UID: \"fa1408e5-0576-4fda-aa6b-830280b670f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" Jul 29 20:56:25.969136 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.969085 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dae9081-9a2d-4867-b0ca-8697d8fe9b77-cert\") pod \"kserve-controller-manager-d6f4b65b8-twx5p\" (UID: \"9dae9081-9a2d-4867-b0ca-8697d8fe9b77\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" Jul 29 20:56:25.969253 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:56:25.969185 2582 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 29 20:56:25.969287 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:56:25.969279 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dae9081-9a2d-4867-b0ca-8697d8fe9b77-cert podName:9dae9081-9a2d-4867-b0ca-8697d8fe9b77 nodeName:}" failed. No retries permitted until 2026-07-29 20:56:26.469258707 +0000 UTC m=+534.592848528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9dae9081-9a2d-4867-b0ca-8697d8fe9b77-cert") pod "kserve-controller-manager-d6f4b65b8-twx5p" (UID: "9dae9081-9a2d-4867-b0ca-8697d8fe9b77") : secret "kserve-webhook-server-cert" not found Jul 29 20:56:25.979162 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:25.979143 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9khfr\" (UniqueName: \"kubernetes.io/projected/9dae9081-9a2d-4867-b0ca-8697d8fe9b77-kube-api-access-9khfr\") pod \"kserve-controller-manager-d6f4b65b8-twx5p\" (UID: \"9dae9081-9a2d-4867-b0ca-8697d8fe9b77\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" Jul 29 20:56:26.070575 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:26.070484 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvfql\" (UniqueName: \"kubernetes.io/projected/fa1408e5-0576-4fda-aa6b-830280b670f3-kube-api-access-hvfql\") pod \"llmisvc-controller-manager-777c6bd7d5-62m6l\" (UID: \"fa1408e5-0576-4fda-aa6b-830280b670f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" Jul 29 20:56:26.070755 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:26.070580 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa1408e5-0576-4fda-aa6b-830280b670f3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-62m6l\" (UID: \"fa1408e5-0576-4fda-aa6b-830280b670f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" Jul 29 20:56:26.070755 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:56:26.070685 2582 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 29 20:56:26.070862 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:56:26.070805 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa1408e5-0576-4fda-aa6b-830280b670f3-cert podName:fa1408e5-0576-4fda-aa6b-830280b670f3 nodeName:}" failed. No retries permitted until 2026-07-29 20:56:26.570781205 +0000 UTC m=+534.694371033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fa1408e5-0576-4fda-aa6b-830280b670f3-cert") pod "llmisvc-controller-manager-777c6bd7d5-62m6l" (UID: "fa1408e5-0576-4fda-aa6b-830280b670f3") : secret "llmisvc-webhook-server-cert" not found Jul 29 20:56:26.081106 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:26.081073 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvfql\" (UniqueName: \"kubernetes.io/projected/fa1408e5-0576-4fda-aa6b-830280b670f3-kube-api-access-hvfql\") pod \"llmisvc-controller-manager-777c6bd7d5-62m6l\" (UID: \"fa1408e5-0576-4fda-aa6b-830280b670f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" Jul 29 20:56:26.475111 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:26.475007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dae9081-9a2d-4867-b0ca-8697d8fe9b77-cert\") pod \"kserve-controller-manager-d6f4b65b8-twx5p\" (UID: \"9dae9081-9a2d-4867-b0ca-8697d8fe9b77\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" Jul 29 20:56:26.477525 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:26.477507 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dae9081-9a2d-4867-b0ca-8697d8fe9b77-cert\") pod \"kserve-controller-manager-d6f4b65b8-twx5p\" (UID: \"9dae9081-9a2d-4867-b0ca-8697d8fe9b77\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" Jul 29 20:56:26.575757 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:26.575724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa1408e5-0576-4fda-aa6b-830280b670f3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-62m6l\" (UID: \"fa1408e5-0576-4fda-aa6b-830280b670f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" Jul 29 20:56:26.578194 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:26.578177 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa1408e5-0576-4fda-aa6b-830280b670f3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-62m6l\" (UID: \"fa1408e5-0576-4fda-aa6b-830280b670f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" Jul 29 20:56:26.717129 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:26.717094 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" Jul 29 20:56:26.742465 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:26.742432 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" Jul 29 20:56:26.860603 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:26.860570 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-twx5p"] Jul 29 20:56:26.863607 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:56:26.863574 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dae9081_9a2d_4867_b0ca_8697d8fe9b77.slice/crio-c3fab116e72573996869a3b4b83d3256e83be1b29c509472103436a98d5b3ccf WatchSource:0}: Error finding container c3fab116e72573996869a3b4b83d3256e83be1b29c509472103436a98d5b3ccf: Status 404 returned error can't find the container with id c3fab116e72573996869a3b4b83d3256e83be1b29c509472103436a98d5b3ccf Jul 29 20:56:26.893393 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:26.893372 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l"] Jul 29 20:56:26.895308 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:56:26.895285 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfa1408e5_0576_4fda_aa6b_830280b670f3.slice/crio-fdde6cf0c18ec9ffee2e9d324496ddfd4730019d86d15d2e2f707b99739641ff WatchSource:0}: Error finding container fdde6cf0c18ec9ffee2e9d324496ddfd4730019d86d15d2e2f707b99739641ff: Status 404 returned error can't find the container with id fdde6cf0c18ec9ffee2e9d324496ddfd4730019d86d15d2e2f707b99739641ff Jul 29 20:56:27.335538 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.335500 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx"] Jul 29 20:56:27.335749 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.335723 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" podUID="533d8e63-4a3d-4c8c-a3ed-fba6cb368623" containerName="discovery" containerID="cri-o://964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26" gracePeriod=30 Jul 29 20:56:27.369214 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.369184 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" event={"ID":"fa1408e5-0576-4fda-aa6b-830280b670f3","Type":"ContainerStarted","Data":"fdde6cf0c18ec9ffee2e9d324496ddfd4730019d86d15d2e2f707b99739641ff"} Jul 29 20:56:27.370375 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.370353 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" event={"ID":"9dae9081-9a2d-4867-b0ca-8697d8fe9b77","Type":"ContainerStarted","Data":"c3fab116e72573996869a3b4b83d3256e83be1b29c509472103436a98d5b3ccf"} Jul 29 20:56:27.586394 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.586339 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:27.686988 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.686956 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-local-certs\") pod \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " Jul 29 20:56:27.687178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.687004 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-csr-dns-cert\") pod \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " Jul 29 20:56:27.687178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.687061 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxv2z\" (UniqueName: \"kubernetes.io/projected/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-kube-api-access-dxv2z\") pod \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " Jul 29 20:56:27.687178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.687088 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-csr-ca-configmap\") pod \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " Jul 29 20:56:27.687178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.687117 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-token\") pod \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " Jul 29 20:56:27.687178 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.687173 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-kubeconfig\") pod \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " Jul 29 20:56:27.687457 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.687230 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-cacerts\") pod \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\" (UID: \"533d8e63-4a3d-4c8c-a3ed-fba6cb368623\") " Jul 29 20:56:27.688203 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.688159 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "533d8e63-4a3d-4c8c-a3ed-fba6cb368623" (UID: "533d8e63-4a3d-4c8c-a3ed-fba6cb368623"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:56:27.690192 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.690163 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-local-certs" (OuterVolumeSpecName: "local-certs") pod "533d8e63-4a3d-4c8c-a3ed-fba6cb368623" (UID: "533d8e63-4a3d-4c8c-a3ed-fba6cb368623"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:56:27.691096 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.691071 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-cacerts" (OuterVolumeSpecName: "cacerts") pod "533d8e63-4a3d-4c8c-a3ed-fba6cb368623" (UID: "533d8e63-4a3d-4c8c-a3ed-fba6cb368623"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:56:27.691206 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.691095 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "533d8e63-4a3d-4c8c-a3ed-fba6cb368623" (UID: "533d8e63-4a3d-4c8c-a3ed-fba6cb368623"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:56:27.691591 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.691562 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-kube-api-access-dxv2z" (OuterVolumeSpecName: "kube-api-access-dxv2z") pod "533d8e63-4a3d-4c8c-a3ed-fba6cb368623" (UID: "533d8e63-4a3d-4c8c-a3ed-fba6cb368623"). InnerVolumeSpecName "kube-api-access-dxv2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:56:27.691783 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.691751 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-token" (OuterVolumeSpecName: "istio-token") pod "533d8e63-4a3d-4c8c-a3ed-fba6cb368623" (UID: "533d8e63-4a3d-4c8c-a3ed-fba6cb368623"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:56:27.692149 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.692114 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "533d8e63-4a3d-4c8c-a3ed-fba6cb368623" (UID: "533d8e63-4a3d-4c8c-a3ed-fba6cb368623"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:56:27.788197 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.788159 2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-local-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:56:27.788197 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.788198 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-csr-dns-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:56:27.788429 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.788215 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxv2z\" (UniqueName: \"kubernetes.io/projected/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-kube-api-access-dxv2z\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:56:27.788429 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.788229 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-csr-ca-configmap\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:56:27.788429 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.788242 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:56:27.788429 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.788256 2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-istio-kubeconfig\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:56:27.788429 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:27.788268 2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/533d8e63-4a3d-4c8c-a3ed-fba6cb368623-cacerts\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:56:28.379555 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:28.379273 2582 generic.go:356] "Generic (PLEG): container finished" podID="533d8e63-4a3d-4c8c-a3ed-fba6cb368623" containerID="964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26" exitCode=0 Jul 29 20:56:28.379555 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:28.379349 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" Jul 29 20:56:28.379555 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:28.379348 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" event={"ID":"533d8e63-4a3d-4c8c-a3ed-fba6cb368623","Type":"ContainerDied","Data":"964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26"} Jul 29 20:56:28.379555 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:28.379401 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx" event={"ID":"533d8e63-4a3d-4c8c-a3ed-fba6cb368623","Type":"ContainerDied","Data":"02624a16579ac9457f7b4a3ba85b0b4293867f0c56ceca21ee9590805d8ffe87"} Jul 29 20:56:28.379555 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:28.379428 2582 scope.go:117] "RemoveContainer" containerID="964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26" Jul 29 20:56:28.406701 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:28.406630 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx"] Jul 29 20:56:28.410006 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:28.409987 2582 scope.go:117] "RemoveContainer" containerID="964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26" Jul 29 20:56:28.410344 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:56:28.410301 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26\": container with ID starting with 964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26 not found: ID does not exist" containerID="964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26" Jul 29 20:56:28.410455 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:28.410347 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26"} err="failed to get container status \"964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26\": rpc error: code = NotFound desc = could not find container \"964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26\": container with ID starting with 964ef7f1fa09f947e6bc23c398b9ce83c5b781e8155a254ab393311ad47b1d26 not found: ID does not exist" Jul 29 20:56:28.413471 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:28.413447 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n4wzx"] Jul 29 20:56:28.493988 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:28.493961 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="533d8e63-4a3d-4c8c-a3ed-fba6cb368623" path="/var/lib/kubelet/pods/533d8e63-4a3d-4c8c-a3ed-fba6cb368623/volumes" Jul 29 20:56:29.386916 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:29.386839 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" event={"ID":"9dae9081-9a2d-4867-b0ca-8697d8fe9b77","Type":"ContainerStarted","Data":"1b381cebdd0660654d704e0db030ca94d25d217a7ee17987348f8f38e70e8e90"} Jul 29 20:56:29.387367 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:29.386979 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" Jul 29 20:56:29.435850 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:29.435797 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" podStartSLOduration=2.182865447 podStartE2EDuration="4.435781615s" totalImagesPullingTime="2.252916168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:29.434202192 +0000 UTC m=+537.557792032" watchObservedRunningTime="2026-07-29 20:56:29.435781615 +0000 UTC m=+537.559371461" Jul 29 20:56:30.392576 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:30.392530 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" event={"ID":"fa1408e5-0576-4fda-aa6b-830280b670f3","Type":"ContainerStarted","Data":"8f5e4ee945f26f155f1fadfe180ac4dc09d149d0f0662204294066c8a3df4367"} Jul 29 20:56:30.408743 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:30.408698 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" podStartSLOduration=2.505099744 podStartE2EDuration="5.408683035s" totalImagesPullingTime="2.903583291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:30.407734603 +0000 UTC m=+538.531324446" watchObservedRunningTime="2026-07-29 20:56:30.408683035 +0000 UTC m=+538.532272876" Jul 29 20:56:31.400920 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:31.398610 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" Jul 29 20:56:32.348302 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:32.348278 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8r9ws" Jul 29 20:56:32.350092 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:32.350073 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-nn9t2" Jul 29 20:56:42.390107 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:42.390061 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 20:56:42.390372 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:42.390143 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 20:56:42.390372 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:42.390178 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 20:56:43.450359 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:43.450326 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" event={"ID":"4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0","Type":"ContainerStarted","Data":"c849b283c68d75028c4a2acf9a29115056a039d590d98d2f3ff14b71aa71efb0"} Jul 29 20:56:43.474159 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:43.474106 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" podStartSLOduration=1.611256799 podStartE2EDuration="19.47409017s" totalImagesPullingTime="17.862833371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:43.472494013 +0000 UTC m=+551.596083856" watchObservedRunningTime="2026-07-29 20:56:43.47409017 +0000 UTC m=+551.597680278" Jul 29 20:56:44.380083 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:44.380046 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:44.384573 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:44.384550 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:44.454537 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:44.454508 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:56:44.455612 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:56:44.455595 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk" Jul 29 20:57:00.398101 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:00.398071 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-twx5p" Jul 29 20:57:02.405368 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:02.405332 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-62m6l" Jul 29 20:57:16.383951 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.383918 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kfx29"] Jul 29 20:57:16.384339 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.384212 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533d8e63-4a3d-4c8c-a3ed-fba6cb368623" containerName="discovery" Jul 29 20:57:16.384339 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.384222 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="533d8e63-4a3d-4c8c-a3ed-fba6cb368623" containerName="discovery" Jul 29 20:57:16.384339 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.384271 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="533d8e63-4a3d-4c8c-a3ed-fba6cb368623" containerName="discovery" Jul 29 20:57:16.396542 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.396519 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kfx29"] Jul 29 20:57:16.396692 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.396605 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kfx29" Jul 29 20:57:16.399114 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.399090 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 20:57:16.400103 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.400078 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-9sfb2\"" Jul 29 20:57:16.400228 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.400084 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 20:57:16.479438 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.479410 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z5r6\" (UniqueName: \"kubernetes.io/projected/ee305eec-c15c-4bbb-8efc-8466e97bb17e-kube-api-access-4z5r6\") pod \"kuadrant-operator-catalog-kfx29\" (UID: \"ee305eec-c15c-4bbb-8efc-8466e97bb17e\") " pod="kuadrant-system/kuadrant-operator-catalog-kfx29" Jul 29 20:57:16.580677 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.580648 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z5r6\" (UniqueName: \"kubernetes.io/projected/ee305eec-c15c-4bbb-8efc-8466e97bb17e-kube-api-access-4z5r6\") pod \"kuadrant-operator-catalog-kfx29\" (UID: \"ee305eec-c15c-4bbb-8efc-8466e97bb17e\") " pod="kuadrant-system/kuadrant-operator-catalog-kfx29" Jul 29 20:57:16.590335 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.590314 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z5r6\" (UniqueName: \"kubernetes.io/projected/ee305eec-c15c-4bbb-8efc-8466e97bb17e-kube-api-access-4z5r6\") pod \"kuadrant-operator-catalog-kfx29\" (UID: \"ee305eec-c15c-4bbb-8efc-8466e97bb17e\") " pod="kuadrant-system/kuadrant-operator-catalog-kfx29" Jul 29 20:57:16.708489 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.708417 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kfx29" Jul 29 20:57:16.743103 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.743075 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kfx29"] Jul 29 20:57:16.836088 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.836060 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kfx29"] Jul 29 20:57:16.839410 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:57:16.839385 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee305eec_c15c_4bbb_8efc_8466e97bb17e.slice/crio-43ae7ff22ce8b0dbe831671ea9a2c0ec4b351b045c38025b11702835b49270e5 WatchSource:0}: Error finding container 43ae7ff22ce8b0dbe831671ea9a2c0ec4b351b045c38025b11702835b49270e5: Status 404 returned error can't find the container with id 43ae7ff22ce8b0dbe831671ea9a2c0ec4b351b045c38025b11702835b49270e5 Jul 29 20:57:16.953816 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.953789 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-m5zzj"] Jul 29 20:57:16.958485 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.958471 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-m5zzj" Jul 29 20:57:16.964458 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:16.964391 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-m5zzj"] Jul 29 20:57:17.084187 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:17.084161 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzjbg\" (UniqueName: \"kubernetes.io/projected/17ec357e-585c-436c-8476-cd3c290a65a9-kube-api-access-gzjbg\") pod \"kuadrant-operator-catalog-m5zzj\" (UID: \"17ec357e-585c-436c-8476-cd3c290a65a9\") " pod="kuadrant-system/kuadrant-operator-catalog-m5zzj" Jul 29 20:57:17.184810 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:17.184784 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzjbg\" (UniqueName: \"kubernetes.io/projected/17ec357e-585c-436c-8476-cd3c290a65a9-kube-api-access-gzjbg\") pod \"kuadrant-operator-catalog-m5zzj\" (UID: \"17ec357e-585c-436c-8476-cd3c290a65a9\") " pod="kuadrant-system/kuadrant-operator-catalog-m5zzj" Jul 29 20:57:17.193616 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:17.193590 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzjbg\" (UniqueName: \"kubernetes.io/projected/17ec357e-585c-436c-8476-cd3c290a65a9-kube-api-access-gzjbg\") pod \"kuadrant-operator-catalog-m5zzj\" (UID: \"17ec357e-585c-436c-8476-cd3c290a65a9\") " pod="kuadrant-system/kuadrant-operator-catalog-m5zzj" Jul 29 20:57:17.267967 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:17.267946 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-m5zzj" Jul 29 20:57:17.402368 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:17.402347 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-m5zzj"] Jul 29 20:57:17.404600 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:57:17.404570 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ec357e_585c_436c_8476_cd3c290a65a9.slice/crio-6542adaa97a85724568ce3e90b3b52cfaed216d004db48b7636aa381e846ea9f WatchSource:0}: Error finding container 6542adaa97a85724568ce3e90b3b52cfaed216d004db48b7636aa381e846ea9f: Status 404 returned error can't find the container with id 6542adaa97a85724568ce3e90b3b52cfaed216d004db48b7636aa381e846ea9f Jul 29 20:57:17.586874 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:17.586782 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kfx29" event={"ID":"ee305eec-c15c-4bbb-8efc-8466e97bb17e","Type":"ContainerStarted","Data":"43ae7ff22ce8b0dbe831671ea9a2c0ec4b351b045c38025b11702835b49270e5"} Jul 29 20:57:17.587995 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:17.587974 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-m5zzj" event={"ID":"17ec357e-585c-436c-8476-cd3c290a65a9","Type":"ContainerStarted","Data":"6542adaa97a85724568ce3e90b3b52cfaed216d004db48b7636aa381e846ea9f"} Jul 29 20:57:19.598957 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:19.598919 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kfx29" event={"ID":"ee305eec-c15c-4bbb-8efc-8466e97bb17e","Type":"ContainerStarted","Data":"8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019"} Jul 29 20:57:19.599454 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:19.598971 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-kfx29" podUID="ee305eec-c15c-4bbb-8efc-8466e97bb17e" containerName="registry-server" containerID="cri-o://8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019" gracePeriod=2 Jul 29 20:57:19.600470 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:19.600448 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-m5zzj" event={"ID":"17ec357e-585c-436c-8476-cd3c290a65a9","Type":"ContainerStarted","Data":"e3aa531dbd58ebfdea28ac05790ed42290a6d0e18c006207f794dcae467ae1c3"} Jul 29 20:57:19.615224 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:19.615186 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kfx29" podStartSLOduration=1.521442993 podStartE2EDuration="3.615173365s" totalImagesPullingTime="2.093730372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:57:19.613331249 +0000 UTC m=+587.736921090" watchObservedRunningTime="2026-07-29 20:57:19.615173365 +0000 UTC m=+587.738763207" Jul 29 20:57:19.628555 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:19.628518 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-m5zzj" podStartSLOduration=2.097686718 podStartE2EDuration="3.628507419s" totalImagesPullingTime="1.530820701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:57:19.627816258 +0000 UTC m=+587.751406099" watchObservedRunningTime="2026-07-29 20:57:19.628507419 +0000 UTC m=+587.752097260" Jul 29 20:57:19.839302 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:19.839280 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kfx29" Jul 29 20:57:19.910790 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:19.910733 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z5r6\" (UniqueName: \"kubernetes.io/projected/ee305eec-c15c-4bbb-8efc-8466e97bb17e-kube-api-access-4z5r6\") pod \"ee305eec-c15c-4bbb-8efc-8466e97bb17e\" (UID: \"ee305eec-c15c-4bbb-8efc-8466e97bb17e\") " Jul 29 20:57:19.912826 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:19.912805 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee305eec-c15c-4bbb-8efc-8466e97bb17e-kube-api-access-4z5r6" (OuterVolumeSpecName: "kube-api-access-4z5r6") pod "ee305eec-c15c-4bbb-8efc-8466e97bb17e" (UID: "ee305eec-c15c-4bbb-8efc-8466e97bb17e"). InnerVolumeSpecName "kube-api-access-4z5r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:57:20.011895 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:20.011871 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z5r6\" (UniqueName: \"kubernetes.io/projected/ee305eec-c15c-4bbb-8efc-8466e97bb17e-kube-api-access-4z5r6\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:20.605081 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:20.605051 2582 generic.go:356] "Generic (PLEG): container finished" podID="ee305eec-c15c-4bbb-8efc-8466e97bb17e" containerID="8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019" exitCode=0 Jul 29 20:57:20.605441 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:20.605076 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kfx29" event={"ID":"ee305eec-c15c-4bbb-8efc-8466e97bb17e","Type":"ContainerDied","Data":"8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019"} Jul 29 20:57:20.605441 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:20.605089 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kfx29" Jul 29 20:57:20.605441 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:20.605134 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kfx29" event={"ID":"ee305eec-c15c-4bbb-8efc-8466e97bb17e","Type":"ContainerDied","Data":"43ae7ff22ce8b0dbe831671ea9a2c0ec4b351b045c38025b11702835b49270e5"} Jul 29 20:57:20.605441 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:20.605155 2582 scope.go:117] "RemoveContainer" containerID="8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019" Jul 29 20:57:20.613768 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:20.613751 2582 scope.go:117] "RemoveContainer" containerID="8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019" Jul 29 20:57:20.614048 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:57:20.614009 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019\": container with ID starting with 8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019 not found: ID does not exist" containerID="8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019" Jul 29 20:57:20.614118 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:20.614056 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019"} err="failed to get container status \"8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019\": rpc error: code = NotFound desc = could not find container \"8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019\": container with ID starting with 8c2cc888d8160d7409d8c76f575a5cf361d253ebf7d937b269825155241a0019 not found: ID does not exist" Jul 29 20:57:20.620475 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:20.620454 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kfx29"] Jul 29 20:57:20.622374 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:20.622356 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kfx29"] Jul 29 20:57:22.491751 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:22.491718 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee305eec-c15c-4bbb-8efc-8466e97bb17e" path="/var/lib/kubelet/pods/ee305eec-c15c-4bbb-8efc-8466e97bb17e/volumes" Jul 29 20:57:27.268546 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:27.268512 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-m5zzj" Jul 29 20:57:27.269014 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:27.268591 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-m5zzj" Jul 29 20:57:27.289788 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:27.289759 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-m5zzj" Jul 29 20:57:27.654958 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:27.654888 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-m5zzj" Jul 29 20:57:31.716195 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.716166 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms"] Jul 29 20:57:31.716654 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.716498 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee305eec-c15c-4bbb-8efc-8466e97bb17e" containerName="registry-server" Jul 29 20:57:31.716654 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.716510 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee305eec-c15c-4bbb-8efc-8466e97bb17e" containerName="registry-server" Jul 29 20:57:31.716654 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.716572 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee305eec-c15c-4bbb-8efc-8466e97bb17e" containerName="registry-server" Jul 29 20:57:31.719405 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.719387 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:31.723142 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.723122 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hlkjq\"" Jul 29 20:57:31.733179 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.733157 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms"] Jul 29 20:57:31.806056 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.806009 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k7zp\" (UniqueName: \"kubernetes.io/projected/41d29d53-b34e-4aa0-9b3e-a313780f825c-kube-api-access-2k7zp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms\" (UID: \"41d29d53-b34e-4aa0-9b3e-a313780f825c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:31.806220 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.806060 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41d29d53-b34e-4aa0-9b3e-a313780f825c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms\" (UID: \"41d29d53-b34e-4aa0-9b3e-a313780f825c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:31.806220 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.806173 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41d29d53-b34e-4aa0-9b3e-a313780f825c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms\" (UID: \"41d29d53-b34e-4aa0-9b3e-a313780f825c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:31.907232 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.907206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41d29d53-b34e-4aa0-9b3e-a313780f825c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms\" (UID: \"41d29d53-b34e-4aa0-9b3e-a313780f825c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:31.907358 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.907261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k7zp\" (UniqueName: \"kubernetes.io/projected/41d29d53-b34e-4aa0-9b3e-a313780f825c-kube-api-access-2k7zp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms\" (UID: \"41d29d53-b34e-4aa0-9b3e-a313780f825c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:31.907358 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.907282 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41d29d53-b34e-4aa0-9b3e-a313780f825c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms\" (UID: \"41d29d53-b34e-4aa0-9b3e-a313780f825c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:31.907613 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.907597 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41d29d53-b34e-4aa0-9b3e-a313780f825c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms\" (UID: \"41d29d53-b34e-4aa0-9b3e-a313780f825c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:31.907673 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.907635 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41d29d53-b34e-4aa0-9b3e-a313780f825c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms\" (UID: \"41d29d53-b34e-4aa0-9b3e-a313780f825c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:31.916277 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:31.916259 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k7zp\" (UniqueName: \"kubernetes.io/projected/41d29d53-b34e-4aa0-9b3e-a313780f825c-kube-api-access-2k7zp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms\" (UID: \"41d29d53-b34e-4aa0-9b3e-a313780f825c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:32.028926 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.028904 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:32.158438 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.158415 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms"] Jul 29 20:57:32.160306 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:57:32.160280 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d29d53_b34e_4aa0_9b3e_a313780f825c.slice/crio-8eafa93bd967d12a6e93c312bbe9f712abd7bb097cf23c4b146f4db06065c55d WatchSource:0}: Error finding container 8eafa93bd967d12a6e93c312bbe9f712abd7bb097cf23c4b146f4db06065c55d: Status 404 returned error can't find the container with id 8eafa93bd967d12a6e93c312bbe9f712abd7bb097cf23c4b146f4db06065c55d Jul 29 20:57:32.331443 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.331341 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm"] Jul 29 20:57:32.335802 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.335786 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:32.343494 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.343475 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm"] Jul 29 20:57:32.411116 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.411087 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm\" (UID: \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:32.411226 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.411155 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjkb\" (UniqueName: \"kubernetes.io/projected/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-kube-api-access-8sjkb\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm\" (UID: \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:32.411226 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.411195 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm\" (UID: \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:32.511909 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.511882 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm\" (UID: \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:32.512058 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.511925 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjkb\" (UniqueName: \"kubernetes.io/projected/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-kube-api-access-8sjkb\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm\" (UID: \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:32.512058 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.512038 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm\" (UID: \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:32.512241 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.512222 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm\" (UID: \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:32.512333 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.512316 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm\" (UID: \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:32.520692 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.520671 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjkb\" (UniqueName: \"kubernetes.io/projected/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-kube-api-access-8sjkb\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm\" (UID: \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:32.645388 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.645312 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:32.660149 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.660114 2582 generic.go:356] "Generic (PLEG): container finished" podID="41d29d53-b34e-4aa0-9b3e-a313780f825c" containerID="2104d4bea0f0a38f4a37ba41fda4f50eb6d36fb392bfaf7df9eb6fc7ad9099cf" exitCode=0 Jul 29 20:57:32.660270 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.660193 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" event={"ID":"41d29d53-b34e-4aa0-9b3e-a313780f825c","Type":"ContainerDied","Data":"2104d4bea0f0a38f4a37ba41fda4f50eb6d36fb392bfaf7df9eb6fc7ad9099cf"} Jul 29 20:57:32.660316 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.660276 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" event={"ID":"41d29d53-b34e-4aa0-9b3e-a313780f825c","Type":"ContainerStarted","Data":"8eafa93bd967d12a6e93c312bbe9f712abd7bb097cf23c4b146f4db06065c55d"} Jul 29 20:57:32.773889 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.773862 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm"] Jul 29 20:57:32.775850 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:57:32.775822 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad3b3a4_b175_45e4_88f2_cce5dc55bb43.slice/crio-d6f91fbc122edd3dca44b86f0340764fbd2b515f3c14e77d1a1bd7fcd5578ff5 WatchSource:0}: Error finding container d6f91fbc122edd3dca44b86f0340764fbd2b515f3c14e77d1a1bd7fcd5578ff5: Status 404 returned error can't find the container with id d6f91fbc122edd3dca44b86f0340764fbd2b515f3c14e77d1a1bd7fcd5578ff5 Jul 29 20:57:32.911469 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.911379 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p"] Jul 29 20:57:32.914818 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.914796 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:32.922462 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:32.922433 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p"] Jul 29 20:57:33.017441 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.017412 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj5gn\" (UniqueName: \"kubernetes.io/projected/bbf76bf4-b259-4481-8b45-4cde2c1a141a-kube-api-access-tj5gn\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p\" (UID: \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:33.017548 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.017454 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbf76bf4-b259-4481-8b45-4cde2c1a141a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p\" (UID: \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:33.017619 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.017560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbf76bf4-b259-4481-8b45-4cde2c1a141a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p\" (UID: \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:33.118222 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.118202 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbf76bf4-b259-4481-8b45-4cde2c1a141a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p\" (UID: \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:33.118320 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.118242 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj5gn\" (UniqueName: \"kubernetes.io/projected/bbf76bf4-b259-4481-8b45-4cde2c1a141a-kube-api-access-tj5gn\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p\" (UID: \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:33.118320 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.118263 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbf76bf4-b259-4481-8b45-4cde2c1a141a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p\" (UID: \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:33.118553 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.118537 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbf76bf4-b259-4481-8b45-4cde2c1a141a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p\" (UID: \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:33.118608 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.118592 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbf76bf4-b259-4481-8b45-4cde2c1a141a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p\" (UID: \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:33.128601 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.128579 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj5gn\" (UniqueName: \"kubernetes.io/projected/bbf76bf4-b259-4481-8b45-4cde2c1a141a-kube-api-access-tj5gn\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p\" (UID: \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:33.254578 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.254553 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:33.313131 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.313105 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj"] Jul 29 20:57:33.317387 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.317365 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:33.324676 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.324644 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj"] Jul 29 20:57:33.390846 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.390823 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p"] Jul 29 20:57:33.392798 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:57:33.392771 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf76bf4_b259_4481_8b45_4cde2c1a141a.slice/crio-1b3de29bb384a56d8073ebfd573d9ae47dbb6350e29411beec67c0368e5c67f6 WatchSource:0}: Error finding container 1b3de29bb384a56d8073ebfd573d9ae47dbb6350e29411beec67c0368e5c67f6: Status 404 returned error can't find the container with id 1b3de29bb384a56d8073ebfd573d9ae47dbb6350e29411beec67c0368e5c67f6 Jul 29 20:57:33.420522 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.420500 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj\" (UID: \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:33.420624 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.420553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhjp6\" (UniqueName: \"kubernetes.io/projected/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-kube-api-access-dhjp6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj\" (UID: \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:33.420624 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.420572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj\" (UID: \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:33.521154 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.521096 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhjp6\" (UniqueName: \"kubernetes.io/projected/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-kube-api-access-dhjp6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj\" (UID: \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:33.521154 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.521132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj\" (UID: \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:33.521287 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.521199 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj\" (UID: \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:33.521584 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.521568 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj\" (UID: \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:33.521621 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.521606 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj\" (UID: \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:33.529916 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.529891 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhjp6\" (UniqueName: \"kubernetes.io/projected/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-kube-api-access-dhjp6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj\" (UID: \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:33.627998 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.627965 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:33.666890 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.666864 2582 generic.go:356] "Generic (PLEG): container finished" podID="8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" containerID="a3a3a26567e4d02b08d0e4e2cc65739e647e04fdd624e840393116c04cbac15c" exitCode=0 Jul 29 20:57:33.667015 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.666990 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" event={"ID":"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43","Type":"ContainerDied","Data":"a3a3a26567e4d02b08d0e4e2cc65739e647e04fdd624e840393116c04cbac15c"} Jul 29 20:57:33.667106 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.667076 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" event={"ID":"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43","Type":"ContainerStarted","Data":"d6f91fbc122edd3dca44b86f0340764fbd2b515f3c14e77d1a1bd7fcd5578ff5"} Jul 29 20:57:33.669828 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.668795 2582 generic.go:356] "Generic (PLEG): container finished" podID="bbf76bf4-b259-4481-8b45-4cde2c1a141a" containerID="307e8731cb688688484596bed4fb4b5cebef0e845045ff4c2121ce9b89ab71b4" exitCode=0 Jul 29 20:57:33.669828 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.668868 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" event={"ID":"bbf76bf4-b259-4481-8b45-4cde2c1a141a","Type":"ContainerDied","Data":"307e8731cb688688484596bed4fb4b5cebef0e845045ff4c2121ce9b89ab71b4"} Jul 29 20:57:33.669828 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.668905 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" event={"ID":"bbf76bf4-b259-4481-8b45-4cde2c1a141a","Type":"ContainerStarted","Data":"1b3de29bb384a56d8073ebfd573d9ae47dbb6350e29411beec67c0368e5c67f6"} Jul 29 20:57:33.769712 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:33.769686 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj"] Jul 29 20:57:33.771880 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:57:33.771828 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3386e07_5cae_4f80_ae1a_179ec7d5ecfa.slice/crio-083c35517750071487ff8ab5f2b53d9f04167e3bd7a20b3e195b075aaf56a677 WatchSource:0}: Error finding container 083c35517750071487ff8ab5f2b53d9f04167e3bd7a20b3e195b075aaf56a677: Status 404 returned error can't find the container with id 083c35517750071487ff8ab5f2b53d9f04167e3bd7a20b3e195b075aaf56a677 Jul 29 20:57:34.676204 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:34.676132 2582 generic.go:356] "Generic (PLEG): container finished" podID="41d29d53-b34e-4aa0-9b3e-a313780f825c" containerID="547f0906e02d4070ba71eb8216075a28c6539883de8b13174834b64dffcca5d5" exitCode=0 Jul 29 20:57:34.676618 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:34.676209 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" event={"ID":"41d29d53-b34e-4aa0-9b3e-a313780f825c","Type":"ContainerDied","Data":"547f0906e02d4070ba71eb8216075a28c6539883de8b13174834b64dffcca5d5"} Jul 29 20:57:34.678095 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:34.677943 2582 generic.go:356] "Generic (PLEG): container finished" podID="d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" containerID="9ae29d3c0840fb5ae3f9d70473861a8a90c30460f2e9e58fb1b5194faa78a95c" exitCode=0 Jul 29 20:57:34.678095 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:34.678061 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" event={"ID":"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa","Type":"ContainerDied","Data":"9ae29d3c0840fb5ae3f9d70473861a8a90c30460f2e9e58fb1b5194faa78a95c"} Jul 29 20:57:34.678212 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:34.678111 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" event={"ID":"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa","Type":"ContainerStarted","Data":"083c35517750071487ff8ab5f2b53d9f04167e3bd7a20b3e195b075aaf56a677"} Jul 29 20:57:34.680056 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:34.680013 2582 generic.go:356] "Generic (PLEG): container finished" podID="8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" containerID="4f75e0e9f15aec1eb69e61d7cd51c0720e3a1934b93d81cde638734896cb45ef" exitCode=0 Jul 29 20:57:34.680168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:34.680061 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" event={"ID":"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43","Type":"ContainerDied","Data":"4f75e0e9f15aec1eb69e61d7cd51c0720e3a1934b93d81cde638734896cb45ef"} Jul 29 20:57:35.323414 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.323374 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-844c59684f-d6x4q"] Jul 29 20:57:35.326724 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.326700 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.329243 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.329218 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 20:57:35.329410 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.329378 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 20:57:35.330372 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.330355 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 20:57:35.330490 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.330454 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fbn88\"" Jul 29 20:57:35.330588 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.330569 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 20:57:35.330708 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.330689 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 20:57:35.330822 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.330810 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 20:57:35.330822 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.330814 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 20:57:35.337501 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.337483 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 20:57:35.340396 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.340376 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-844c59684f-d6x4q"] Jul 29 20:57:35.438754 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.438719 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-console-serving-cert\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.438754 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.438751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-oauth-serving-cert\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.438980 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.438775 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-service-ca\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.438980 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.438817 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-console-config\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.438980 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.438865 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9vv\" (UniqueName: \"kubernetes.io/projected/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-kube-api-access-df9vv\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.438980 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.438974 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-console-oauth-config\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.439189 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.439052 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-trusted-ca-bundle\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.540145 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.540119 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-service-ca\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.540274 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.540152 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-console-config\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.540274 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.540178 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9vv\" (UniqueName: \"kubernetes.io/projected/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-kube-api-access-df9vv\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.540274 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.540237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-console-oauth-config\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.540274 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.540263 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-trusted-ca-bundle\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.540488 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.540469 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-console-serving-cert\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.540584 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.540506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-oauth-serving-cert\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.541017 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.540992 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-console-config\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.541153 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.540996 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-service-ca\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.541277 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.541258 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-trusted-ca-bundle\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.541334 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.541318 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-oauth-serving-cert\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.542743 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.542723 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-console-oauth-config\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.542993 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.542975 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-console-serving-cert\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.548743 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.548722 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9vv\" (UniqueName: \"kubernetes.io/projected/aa245b0e-7f9e-4177-81e0-a8b6815f5e1b-kube-api-access-df9vv\") pod \"console-844c59684f-d6x4q\" (UID: \"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b\") " pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.637150 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.637076 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:35.688149 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.688120 2582 generic.go:356] "Generic (PLEG): container finished" podID="8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" containerID="9d96df5c112f3b7545522b8e74620c7dce14258c883b780759aefac9d558fc95" exitCode=0 Jul 29 20:57:35.688506 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.688201 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" event={"ID":"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43","Type":"ContainerDied","Data":"9d96df5c112f3b7545522b8e74620c7dce14258c883b780759aefac9d558fc95"} Jul 29 20:57:35.690320 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.690300 2582 generic.go:356] "Generic (PLEG): container finished" podID="bbf76bf4-b259-4481-8b45-4cde2c1a141a" containerID="432dfa4a3264dd4909ffbf9e8e5d44d889b643a44508107098db18bc09d08f5c" exitCode=0 Jul 29 20:57:35.690426 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.690381 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" event={"ID":"bbf76bf4-b259-4481-8b45-4cde2c1a141a","Type":"ContainerDied","Data":"432dfa4a3264dd4909ffbf9e8e5d44d889b643a44508107098db18bc09d08f5c"} Jul 29 20:57:35.693469 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.693445 2582 generic.go:356] "Generic (PLEG): container finished" podID="41d29d53-b34e-4aa0-9b3e-a313780f825c" containerID="0a2efafff6a3deb6931c3a8b2a4200115c6d46754949c9435743e2f7140285b4" exitCode=0 Jul 29 20:57:35.693566 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.693521 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" event={"ID":"41d29d53-b34e-4aa0-9b3e-a313780f825c","Type":"ContainerDied","Data":"0a2efafff6a3deb6931c3a8b2a4200115c6d46754949c9435743e2f7140285b4"} Jul 29 20:57:35.802464 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:35.801897 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-844c59684f-d6x4q"] Jul 29 20:57:35.806467 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:57:35.806165 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa245b0e_7f9e_4177_81e0_a8b6815f5e1b.slice/crio-feefaccb8204048d15dcb453e615dc34988bc78b7c4d25fb150e50700d98e429 WatchSource:0}: Error finding container feefaccb8204048d15dcb453e615dc34988bc78b7c4d25fb150e50700d98e429: Status 404 returned error can't find the container with id feefaccb8204048d15dcb453e615dc34988bc78b7c4d25fb150e50700d98e429 Jul 29 20:57:36.699475 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.699433 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844c59684f-d6x4q" event={"ID":"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b","Type":"ContainerStarted","Data":"33d01848f307bb3b344269fbe65e6d31adc5964baaa258cea7ff669a3182fa80"} Jul 29 20:57:36.699911 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.699490 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844c59684f-d6x4q" event={"ID":"aa245b0e-7f9e-4177-81e0-a8b6815f5e1b","Type":"ContainerStarted","Data":"feefaccb8204048d15dcb453e615dc34988bc78b7c4d25fb150e50700d98e429"} Jul 29 20:57:36.701487 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.701462 2582 generic.go:356] "Generic (PLEG): container finished" podID="bbf76bf4-b259-4481-8b45-4cde2c1a141a" containerID="b12a400c1e5afa2aa9f8ea0cbd127ced29693a573edb1cd21eec01583678723a" exitCode=0 Jul 29 20:57:36.701592 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.701547 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" event={"ID":"bbf76bf4-b259-4481-8b45-4cde2c1a141a","Type":"ContainerDied","Data":"b12a400c1e5afa2aa9f8ea0cbd127ced29693a573edb1cd21eec01583678723a"} Jul 29 20:57:36.703259 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.703239 2582 generic.go:356] "Generic (PLEG): container finished" podID="d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" containerID="24f2a841323ed69b9c1814d64d4266ecd8529109f183a54566396085e27d1ee4" exitCode=0 Jul 29 20:57:36.703371 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.703303 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" event={"ID":"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa","Type":"ContainerDied","Data":"24f2a841323ed69b9c1814d64d4266ecd8529109f183a54566396085e27d1ee4"} Jul 29 20:57:36.716132 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.716086 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-844c59684f-d6x4q" podStartSLOduration=1.716070068 podStartE2EDuration="1.716070068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:57:36.715757249 +0000 UTC m=+604.839347092" watchObservedRunningTime="2026-07-29 20:57:36.716070068 +0000 UTC m=+604.839659910" Jul 29 20:57:36.839911 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.839888 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:36.897833 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.897810 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:36.953137 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.953084 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sjkb\" (UniqueName: \"kubernetes.io/projected/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-kube-api-access-8sjkb\") pod \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\" (UID: \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\") " Jul 29 20:57:36.953230 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.953142 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-bundle\") pod \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\" (UID: \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\") " Jul 29 20:57:36.953230 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.953184 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-util\") pod \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\" (UID: \"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43\") " Jul 29 20:57:36.953230 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.953200 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41d29d53-b34e-4aa0-9b3e-a313780f825c-util\") pod \"41d29d53-b34e-4aa0-9b3e-a313780f825c\" (UID: \"41d29d53-b34e-4aa0-9b3e-a313780f825c\") " Jul 29 20:57:36.953379 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.953242 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k7zp\" (UniqueName: \"kubernetes.io/projected/41d29d53-b34e-4aa0-9b3e-a313780f825c-kube-api-access-2k7zp\") pod \"41d29d53-b34e-4aa0-9b3e-a313780f825c\" (UID: \"41d29d53-b34e-4aa0-9b3e-a313780f825c\") " Jul 29 20:57:36.953379 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.953290 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41d29d53-b34e-4aa0-9b3e-a313780f825c-bundle\") pod \"41d29d53-b34e-4aa0-9b3e-a313780f825c\" (UID: \"41d29d53-b34e-4aa0-9b3e-a313780f825c\") " Jul 29 20:57:36.953958 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.953933 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-bundle" (OuterVolumeSpecName: "bundle") pod "8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" (UID: "8ad3b3a4-b175-45e4-88f2-cce5dc55bb43"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:57:36.954102 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.953930 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d29d53-b34e-4aa0-9b3e-a313780f825c-bundle" (OuterVolumeSpecName: "bundle") pod "41d29d53-b34e-4aa0-9b3e-a313780f825c" (UID: "41d29d53-b34e-4aa0-9b3e-a313780f825c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:57:36.955303 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.955276 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-kube-api-access-8sjkb" (OuterVolumeSpecName: "kube-api-access-8sjkb") pod "8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" (UID: "8ad3b3a4-b175-45e4-88f2-cce5dc55bb43"). InnerVolumeSpecName "kube-api-access-8sjkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:57:36.955815 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.955791 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d29d53-b34e-4aa0-9b3e-a313780f825c-kube-api-access-2k7zp" (OuterVolumeSpecName: "kube-api-access-2k7zp") pod "41d29d53-b34e-4aa0-9b3e-a313780f825c" (UID: "41d29d53-b34e-4aa0-9b3e-a313780f825c"). InnerVolumeSpecName "kube-api-access-2k7zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:57:36.958961 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.958921 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-util" (OuterVolumeSpecName: "util") pod "8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" (UID: "8ad3b3a4-b175-45e4-88f2-cce5dc55bb43"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:57:36.959642 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:36.959623 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d29d53-b34e-4aa0-9b3e-a313780f825c-util" (OuterVolumeSpecName: "util") pod "41d29d53-b34e-4aa0-9b3e-a313780f825c" (UID: "41d29d53-b34e-4aa0-9b3e-a313780f825c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:57:37.054252 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.054228 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k7zp\" (UniqueName: \"kubernetes.io/projected/41d29d53-b34e-4aa0-9b3e-a313780f825c-kube-api-access-2k7zp\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:37.054252 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.054251 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41d29d53-b34e-4aa0-9b3e-a313780f825c-bundle\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:37.054381 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.054262 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sjkb\" (UniqueName: \"kubernetes.io/projected/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-kube-api-access-8sjkb\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:37.054381 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.054271 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-bundle\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:37.054381 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.054279 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ad3b3a4-b175-45e4-88f2-cce5dc55bb43-util\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:37.054381 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.054287 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41d29d53-b34e-4aa0-9b3e-a313780f825c-util\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:37.709575 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.709545 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" Jul 29 20:57:37.710060 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.709573 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm" event={"ID":"8ad3b3a4-b175-45e4-88f2-cce5dc55bb43","Type":"ContainerDied","Data":"d6f91fbc122edd3dca44b86f0340764fbd2b515f3c14e77d1a1bd7fcd5578ff5"} Jul 29 20:57:37.710060 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.709621 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6f91fbc122edd3dca44b86f0340764fbd2b515f3c14e77d1a1bd7fcd5578ff5" Jul 29 20:57:37.711585 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.711560 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" Jul 29 20:57:37.711695 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.711554 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms" event={"ID":"41d29d53-b34e-4aa0-9b3e-a313780f825c","Type":"ContainerDied","Data":"8eafa93bd967d12a6e93c312bbe9f712abd7bb097cf23c4b146f4db06065c55d"} Jul 29 20:57:37.711695 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.711667 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eafa93bd967d12a6e93c312bbe9f712abd7bb097cf23c4b146f4db06065c55d" Jul 29 20:57:37.713766 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.713739 2582 generic.go:356] "Generic (PLEG): container finished" podID="d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" containerID="6e7074af921d9a792b57fe33efb0c03b09bc29ef011d88d9f5f2d838a58423cd" exitCode=0 Jul 29 20:57:37.713882 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.713765 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" event={"ID":"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa","Type":"ContainerDied","Data":"6e7074af921d9a792b57fe33efb0c03b09bc29ef011d88d9f5f2d838a58423cd"} Jul 29 20:57:37.841078 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.841047 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:37.962308 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.962230 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbf76bf4-b259-4481-8b45-4cde2c1a141a-util\") pod \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\" (UID: \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\") " Jul 29 20:57:37.962308 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.962274 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbf76bf4-b259-4481-8b45-4cde2c1a141a-bundle\") pod \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\" (UID: \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\") " Jul 29 20:57:37.962509 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.962313 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj5gn\" (UniqueName: \"kubernetes.io/projected/bbf76bf4-b259-4481-8b45-4cde2c1a141a-kube-api-access-tj5gn\") pod \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\" (UID: \"bbf76bf4-b259-4481-8b45-4cde2c1a141a\") " Jul 29 20:57:37.962707 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.962681 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf76bf4-b259-4481-8b45-4cde2c1a141a-bundle" (OuterVolumeSpecName: "bundle") pod "bbf76bf4-b259-4481-8b45-4cde2c1a141a" (UID: "bbf76bf4-b259-4481-8b45-4cde2c1a141a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:57:37.964486 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.964465 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf76bf4-b259-4481-8b45-4cde2c1a141a-kube-api-access-tj5gn" (OuterVolumeSpecName: "kube-api-access-tj5gn") pod "bbf76bf4-b259-4481-8b45-4cde2c1a141a" (UID: "bbf76bf4-b259-4481-8b45-4cde2c1a141a"). InnerVolumeSpecName "kube-api-access-tj5gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:57:37.970304 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:37.970280 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf76bf4-b259-4481-8b45-4cde2c1a141a-util" (OuterVolumeSpecName: "util") pod "bbf76bf4-b259-4481-8b45-4cde2c1a141a" (UID: "bbf76bf4-b259-4481-8b45-4cde2c1a141a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:57:38.063334 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.063296 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbf76bf4-b259-4481-8b45-4cde2c1a141a-util\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:38.063334 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.063329 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbf76bf4-b259-4481-8b45-4cde2c1a141a-bundle\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:38.063334 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.063341 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj5gn\" (UniqueName: \"kubernetes.io/projected/bbf76bf4-b259-4481-8b45-4cde2c1a141a-kube-api-access-tj5gn\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:38.721057 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.721003 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" event={"ID":"bbf76bf4-b259-4481-8b45-4cde2c1a141a","Type":"ContainerDied","Data":"1b3de29bb384a56d8073ebfd573d9ae47dbb6350e29411beec67c0368e5c67f6"} Jul 29 20:57:38.721424 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.721074 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b3de29bb384a56d8073ebfd573d9ae47dbb6350e29411beec67c0368e5c67f6" Jul 29 20:57:38.721424 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.721197 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p" Jul 29 20:57:38.856140 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.856114 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:38.970531 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.970507 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-util\") pod \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\" (UID: \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\") " Jul 29 20:57:38.970717 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.970539 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-bundle\") pod \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\" (UID: \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\") " Jul 29 20:57:38.970717 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.970567 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhjp6\" (UniqueName: \"kubernetes.io/projected/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-kube-api-access-dhjp6\") pod \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\" (UID: \"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa\") " Jul 29 20:57:38.971070 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.971019 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-bundle" (OuterVolumeSpecName: "bundle") pod "d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" (UID: "d3386e07-5cae-4f80-ae1a-179ec7d5ecfa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:57:38.972741 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.972720 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-kube-api-access-dhjp6" (OuterVolumeSpecName: "kube-api-access-dhjp6") pod "d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" (UID: "d3386e07-5cae-4f80-ae1a-179ec7d5ecfa"). InnerVolumeSpecName "kube-api-access-dhjp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:57:38.978510 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:38.978487 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-util" (OuterVolumeSpecName: "util") pod "d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" (UID: "d3386e07-5cae-4f80-ae1a-179ec7d5ecfa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:57:39.071360 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:39.071302 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhjp6\" (UniqueName: \"kubernetes.io/projected/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-kube-api-access-dhjp6\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:39.071360 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:39.071322 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-util\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:39.071360 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:39.071333 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3386e07-5cae-4f80-ae1a-179ec7d5ecfa-bundle\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:57:39.727323 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:39.727291 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" event={"ID":"d3386e07-5cae-4f80-ae1a-179ec7d5ecfa","Type":"ContainerDied","Data":"083c35517750071487ff8ab5f2b53d9f04167e3bd7a20b3e195b075aaf56a677"} Jul 29 20:57:39.727723 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:39.727360 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj" Jul 29 20:57:39.727723 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:39.727370 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="083c35517750071487ff8ab5f2b53d9f04167e3bd7a20b3e195b075aaf56a677" Jul 29 20:57:45.637594 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:45.637560 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:45.637999 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:45.637619 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:45.642441 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:45.642421 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:45.756666 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:45.756638 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-844c59684f-d6x4q" Jul 29 20:57:48.025750 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.025716 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66"] Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.025999 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbf76bf4-b259-4481-8b45-4cde2c1a141a" containerName="extract" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026009 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbf76bf4-b259-4481-8b45-4cde2c1a141a" containerName="extract" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026019 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" containerName="extract" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026038 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" containerName="extract" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026048 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" containerName="util" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026054 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" containerName="util" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026061 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbf76bf4-b259-4481-8b45-4cde2c1a141a" containerName="util" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026066 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbf76bf4-b259-4481-8b45-4cde2c1a141a" containerName="util" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026073 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" containerName="pull" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026078 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" containerName="pull" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026086 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d29d53-b34e-4aa0-9b3e-a313780f825c" containerName="pull" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026091 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d29d53-b34e-4aa0-9b3e-a313780f825c" containerName="pull" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026101 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" containerName="extract" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026107 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" containerName="extract" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026111 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbf76bf4-b259-4481-8b45-4cde2c1a141a" containerName="pull" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026116 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbf76bf4-b259-4481-8b45-4cde2c1a141a" containerName="pull" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026123 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" containerName="pull" Jul 29 20:57:48.026127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026131 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" containerName="pull" Jul 29 20:57:48.026690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026142 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" containerName="util" Jul 29 20:57:48.026690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026147 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" containerName="util" Jul 29 20:57:48.026690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026153 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d29d53-b34e-4aa0-9b3e-a313780f825c" containerName="extract" Jul 29 20:57:48.026690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026158 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d29d53-b34e-4aa0-9b3e-a313780f825c" containerName="extract" Jul 29 20:57:48.026690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026164 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d29d53-b34e-4aa0-9b3e-a313780f825c" containerName="util" Jul 29 20:57:48.026690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026169 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d29d53-b34e-4aa0-9b3e-a313780f825c" containerName="util" Jul 29 20:57:48.026690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026214 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d29d53-b34e-4aa0-9b3e-a313780f825c" containerName="extract" Jul 29 20:57:48.026690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026221 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ad3b3a4-b175-45e4-88f2-cce5dc55bb43" containerName="extract" Jul 29 20:57:48.026690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026226 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3386e07-5cae-4f80-ae1a-179ec7d5ecfa" containerName="extract" Jul 29 20:57:48.026690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.026232 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbf76bf4-b259-4481-8b45-4cde2c1a141a" containerName="extract" Jul 29 20:57:48.032682 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.032665 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66" Jul 29 20:57:48.035250 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.035226 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jg799\"" Jul 29 20:57:48.035388 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.035373 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 29 20:57:48.039744 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.039559 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66"] Jul 29 20:57:48.143856 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.143826 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9jww\" (UniqueName: \"kubernetes.io/projected/1a865267-c588-4d02-a253-fa7c0c84b099-kube-api-access-f9jww\") pod \"dns-operator-controller-manager-864dd78c4c-dbw66\" (UID: \"1a865267-c588-4d02-a253-fa7c0c84b099\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66" Jul 29 20:57:48.245194 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.245165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9jww\" (UniqueName: \"kubernetes.io/projected/1a865267-c588-4d02-a253-fa7c0c84b099-kube-api-access-f9jww\") pod \"dns-operator-controller-manager-864dd78c4c-dbw66\" (UID: \"1a865267-c588-4d02-a253-fa7c0c84b099\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66" Jul 29 20:57:48.259817 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.259788 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9jww\" (UniqueName: \"kubernetes.io/projected/1a865267-c588-4d02-a253-fa7c0c84b099-kube-api-access-f9jww\") pod \"dns-operator-controller-manager-864dd78c4c-dbw66\" (UID: \"1a865267-c588-4d02-a253-fa7c0c84b099\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66" Jul 29 20:57:48.343633 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.343577 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66" Jul 29 20:57:48.480427 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.480406 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66"] Jul 29 20:57:48.482267 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:57:48.482241 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a865267_c588_4d02_a253_fa7c0c84b099.slice/crio-b45577614e906cebfd3ce9c14c776d0889a3528c7c18aa41462fe0f507111d7e WatchSource:0}: Error finding container b45577614e906cebfd3ce9c14c776d0889a3528c7c18aa41462fe0f507111d7e: Status 404 returned error can't find the container with id b45577614e906cebfd3ce9c14c776d0889a3528c7c18aa41462fe0f507111d7e Jul 29 20:57:48.765840 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:48.765811 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66" event={"ID":"1a865267-c588-4d02-a253-fa7c0c84b099","Type":"ContainerStarted","Data":"b45577614e906cebfd3ce9c14c776d0889a3528c7c18aa41462fe0f507111d7e"} Jul 29 20:57:50.782340 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:50.782240 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66" event={"ID":"1a865267-c588-4d02-a253-fa7c0c84b099","Type":"ContainerStarted","Data":"833d765fed9d03087b38a40fdbfeb0465f0d5d65c4c6cdf7d853fec50c1a4055"} Jul 29 20:57:50.782717 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:50.782409 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66" Jul 29 20:57:50.799052 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:50.798997 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66" podStartSLOduration=0.753724523 podStartE2EDuration="2.798982568s" totalImagesPullingTime="2.045258045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:57:50.797152501 +0000 UTC m=+618.920742345" watchObservedRunningTime="2026-07-29 20:57:50.798982568 +0000 UTC m=+618.922572410" Jul 29 20:57:53.436037 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:53.435946 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s"] Jul 29 20:57:53.441741 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:53.441725 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:57:53.446859 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:53.446835 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wdf79\"" Jul 29 20:57:53.465121 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:53.465102 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s"] Jul 29 20:57:53.591540 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:53.591503 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/efa0e58f-36fa-4038-acf8-6dc90e493350-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" (UID: \"efa0e58f-36fa-4038-acf8-6dc90e493350\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:57:53.591687 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:53.591580 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvcxz\" (UniqueName: \"kubernetes.io/projected/efa0e58f-36fa-4038-acf8-6dc90e493350-kube-api-access-bvcxz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" (UID: \"efa0e58f-36fa-4038-acf8-6dc90e493350\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:57:53.692012 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:53.691937 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvcxz\" (UniqueName: \"kubernetes.io/projected/efa0e58f-36fa-4038-acf8-6dc90e493350-kube-api-access-bvcxz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" (UID: \"efa0e58f-36fa-4038-acf8-6dc90e493350\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:57:53.692191 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:53.692082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/efa0e58f-36fa-4038-acf8-6dc90e493350-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" (UID: \"efa0e58f-36fa-4038-acf8-6dc90e493350\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:57:53.692461 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:53.692442 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/efa0e58f-36fa-4038-acf8-6dc90e493350-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" (UID: \"efa0e58f-36fa-4038-acf8-6dc90e493350\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:57:53.703514 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:53.703492 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvcxz\" (UniqueName: \"kubernetes.io/projected/efa0e58f-36fa-4038-acf8-6dc90e493350-kube-api-access-bvcxz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" (UID: \"efa0e58f-36fa-4038-acf8-6dc90e493350\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:57:53.751123 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:53.751090 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:57:53.897315 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:53.897288 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s"] Jul 29 20:57:53.899707 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:57:53.899677 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa0e58f_36fa_4038_acf8_6dc90e493350.slice/crio-1897bf7e5728c68e3139f6454d68821be04238a61da8654d1656b5fb5747d9d6 WatchSource:0}: Error finding container 1897bf7e5728c68e3139f6454d68821be04238a61da8654d1656b5fb5747d9d6: Status 404 returned error can't find the container with id 1897bf7e5728c68e3139f6454d68821be04238a61da8654d1656b5fb5747d9d6 Jul 29 20:57:54.806067 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:54.806037 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" event={"ID":"efa0e58f-36fa-4038-acf8-6dc90e493350","Type":"ContainerStarted","Data":"1897bf7e5728c68e3139f6454d68821be04238a61da8654d1656b5fb5747d9d6"} Jul 29 20:57:56.698651 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:56.698619 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw"] Jul 29 20:57:56.701894 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:56.701877 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw" Jul 29 20:57:56.704433 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:56.704416 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4fzbp\"" Jul 29 20:57:56.714616 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:56.714594 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw"] Jul 29 20:57:56.817259 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:56.817230 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skhnb\" (UniqueName: \"kubernetes.io/projected/3f8958a3-9c15-4a28-8720-0d4ab0cc2988-kube-api-access-skhnb\") pod \"authorino-operator-75cc8d5cb7-dp5xw\" (UID: \"3f8958a3-9c15-4a28-8720-0d4ab0cc2988\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw" Jul 29 20:57:56.918167 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:56.918130 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhnb\" (UniqueName: \"kubernetes.io/projected/3f8958a3-9c15-4a28-8720-0d4ab0cc2988-kube-api-access-skhnb\") pod \"authorino-operator-75cc8d5cb7-dp5xw\" (UID: \"3f8958a3-9c15-4a28-8720-0d4ab0cc2988\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw" Jul 29 20:57:56.930000 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:56.929972 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhnb\" (UniqueName: \"kubernetes.io/projected/3f8958a3-9c15-4a28-8720-0d4ab0cc2988-kube-api-access-skhnb\") pod \"authorino-operator-75cc8d5cb7-dp5xw\" (UID: \"3f8958a3-9c15-4a28-8720-0d4ab0cc2988\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw" Jul 29 20:57:57.012472 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:57.012441 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw" Jul 29 20:57:57.149140 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:57.149117 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw"] Jul 29 20:57:57.151152 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:57:57.151123 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8958a3_9c15_4a28_8720_0d4ab0cc2988.slice/crio-a7d47864f29161456e36d643460c45b4e792d33187e5182a2b36015bf614b53f WatchSource:0}: Error finding container a7d47864f29161456e36d643460c45b4e792d33187e5182a2b36015bf614b53f: Status 404 returned error can't find the container with id a7d47864f29161456e36d643460c45b4e792d33187e5182a2b36015bf614b53f Jul 29 20:57:57.823517 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:57.823483 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw" event={"ID":"3f8958a3-9c15-4a28-8720-0d4ab0cc2988","Type":"ContainerStarted","Data":"a7d47864f29161456e36d643460c45b4e792d33187e5182a2b36015bf614b53f"} Jul 29 20:57:59.691772 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:59.691735 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9"] Jul 29 20:57:59.695493 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:59.695473 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:57:59.698508 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:59.698486 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qz554\"" Jul 29 20:57:59.707437 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:59.707417 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9"] Jul 29 20:57:59.840049 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:59.839995 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnm8l\" (UniqueName: \"kubernetes.io/projected/05eed357-d4ae-409a-ac02-2830e53d7099-kube-api-access-wnm8l\") pod \"limitador-operator-controller-manager-859d7b78bd-ctkg9\" (UID: \"05eed357-d4ae-409a-ac02-2830e53d7099\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:57:59.941011 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:59.940975 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnm8l\" (UniqueName: \"kubernetes.io/projected/05eed357-d4ae-409a-ac02-2830e53d7099-kube-api-access-wnm8l\") pod \"limitador-operator-controller-manager-859d7b78bd-ctkg9\" (UID: \"05eed357-d4ae-409a-ac02-2830e53d7099\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:57:59.951056 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:57:59.950984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnm8l\" (UniqueName: \"kubernetes.io/projected/05eed357-d4ae-409a-ac02-2830e53d7099-kube-api-access-wnm8l\") pod \"limitador-operator-controller-manager-859d7b78bd-ctkg9\" (UID: \"05eed357-d4ae-409a-ac02-2830e53d7099\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:00.010399 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:00.010363 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:00.700894 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:00.700867 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9"] Jul 29 20:58:00.702610 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:58:00.702581 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05eed357_d4ae_409a_ac02_2830e53d7099.slice/crio-57037c99381b2e629dcb19645614d1aef5a9d829ce40b74bd625dcfea85f466b WatchSource:0}: Error finding container 57037c99381b2e629dcb19645614d1aef5a9d829ce40b74bd625dcfea85f466b: Status 404 returned error can't find the container with id 57037c99381b2e629dcb19645614d1aef5a9d829ce40b74bd625dcfea85f466b Jul 29 20:58:00.840570 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:00.840485 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" event={"ID":"efa0e58f-36fa-4038-acf8-6dc90e493350","Type":"ContainerStarted","Data":"13ac81858bf8c8768c867f3bd40488a960c93112518ee3356c4b50c21edfe9cd"} Jul 29 20:58:00.840731 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:00.840588 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:58:00.842229 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:00.842193 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw" event={"ID":"3f8958a3-9c15-4a28-8720-0d4ab0cc2988","Type":"ContainerStarted","Data":"10a06cbef4d9f7b24773dc3625811b6e7c9a8778aecd2e60ae6c628ce4d6525d"} Jul 29 20:58:00.842352 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:00.842267 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw" Jul 29 20:58:00.843432 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:00.843413 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" event={"ID":"05eed357-d4ae-409a-ac02-2830e53d7099","Type":"ContainerStarted","Data":"57037c99381b2e629dcb19645614d1aef5a9d829ce40b74bd625dcfea85f466b"} Jul 29 20:58:00.860593 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:00.860549 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" podStartSLOduration=1.200355582 podStartE2EDuration="7.860537826s" totalImagesPullingTime="6.660182244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:58:00.857981244 +0000 UTC m=+628.981571089" watchObservedRunningTime="2026-07-29 20:58:00.860537826 +0000 UTC m=+628.984127667" Jul 29 20:58:00.878319 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:00.878277 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw" podStartSLOduration=1.487476365 podStartE2EDuration="4.878265274s" totalImagesPullingTime="3.390788909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:58:00.876529616 +0000 UTC m=+629.000119460" watchObservedRunningTime="2026-07-29 20:58:00.878265274 +0000 UTC m=+629.001855115" Jul 29 20:58:01.789633 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:01.789602 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dbw66" Jul 29 20:58:02.854627 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:02.854590 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" event={"ID":"05eed357-d4ae-409a-ac02-2830e53d7099","Type":"ContainerStarted","Data":"13e35796c47e6f3095f96ffa7c5a66512d0fc2c9f24bee3e711e211a57b1ba47"} Jul 29 20:58:02.855080 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:02.854657 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:02.873687 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:02.873645 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" podStartSLOduration=2.282689806 podStartE2EDuration="3.873632614s" totalImagesPullingTime="1.590942808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:58:02.872221737 +0000 UTC m=+630.995811579" watchObservedRunningTime="2026-07-29 20:58:02.873632614 +0000 UTC m=+630.997222456" Jul 29 20:58:11.850630 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:11.850602 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:58:11.850988 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:11.850668 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dp5xw" Jul 29 20:58:13.444493 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.444458 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q"] Jul 29 20:58:13.449246 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.449230 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.459084 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.459058 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q"] Jul 29 20:58:13.546419 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.546392 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6zxq\" (UniqueName: \"kubernetes.io/projected/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934-kube-api-access-b6zxq\") pod \"kuadrant-operator-controller-manager-55467d9f7-c6f8q\" (UID: \"0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.546681 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.546657 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-c6f8q\" (UID: \"0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.548655 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.548623 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q"] Jul 29 20:58:13.548824 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.548808 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-b6zxq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" podUID="0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934" Jul 29 20:58:13.588242 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.588217 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq"] Jul 29 20:58:13.591390 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.591372 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:13.607876 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.607856 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq"] Jul 29 20:58:13.647815 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.647789 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6zxq\" (UniqueName: \"kubernetes.io/projected/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934-kube-api-access-b6zxq\") pod \"kuadrant-operator-controller-manager-55467d9f7-c6f8q\" (UID: \"0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.647993 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.647851 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72024507-e8e3-479d-9add-3a9bb6c2f93c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9hchq\" (UID: \"72024507-e8e3-479d-9add-3a9bb6c2f93c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:13.647993 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.647886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-c6f8q\" (UID: \"0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.647993 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.647914 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzr5k\" (UniqueName: \"kubernetes.io/projected/72024507-e8e3-479d-9add-3a9bb6c2f93c-kube-api-access-fzr5k\") pod \"kuadrant-operator-controller-manager-86878c769f-9hchq\" (UID: \"72024507-e8e3-479d-9add-3a9bb6c2f93c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:13.648300 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.648282 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-c6f8q\" (UID: \"0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.654103 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.654074 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q"] Jul 29 20:58:13.658070 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.658052 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6zxq\" (UniqueName: \"kubernetes.io/projected/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934-kube-api-access-b6zxq\") pod \"kuadrant-operator-controller-manager-55467d9f7-c6f8q\" (UID: \"0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.659254 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.659232 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q"] Jul 29 20:58:13.669213 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.669191 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s"] Jul 29 20:58:13.669396 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.669373 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" containerName="manager" containerID="cri-o://13ac81858bf8c8768c867f3bd40488a960c93112518ee3356c4b50c21edfe9cd" gracePeriod=2 Jul 29 20:58:13.678065 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.678012 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s"] Jul 29 20:58:13.693500 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.693474 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq"] Jul 29 20:58:13.693827 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.693801 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-fzr5k], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" podUID="72024507-e8e3-479d-9add-3a9bb6c2f93c" Jul 29 20:58:13.695447 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.695397 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq"] Jul 29 20:58:13.699502 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.699480 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg"] Jul 29 20:58:13.699981 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.699961 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" containerName="manager" Jul 29 20:58:13.700065 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.699987 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" containerName="manager" Jul 29 20:58:13.700110 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.700098 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" containerName="manager" Jul 29 20:58:13.703325 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.703307 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" Jul 29 20:58:13.714388 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.714365 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9"] Jul 29 20:58:13.714603 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.714579 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" podUID="05eed357-d4ae-409a-ac02-2830e53d7099" containerName="manager" containerID="cri-o://13e35796c47e6f3095f96ffa7c5a66512d0fc2c9f24bee3e711e211a57b1ba47" gracePeriod=2 Jul 29 20:58:13.716747 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.716715 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:13.719338 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.719312 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg"] Jul 29 20:58:13.736695 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.736661 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:58:13.736880 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.736859 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9"] Jul 29 20:58:13.738874 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.738849 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ctkg9\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="05eed357-d4ae-409a-ac02-2830e53d7099" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:13.739687 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.739669 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx"] Jul 29 20:58:13.740121 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.740104 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05eed357-d4ae-409a-ac02-2830e53d7099" containerName="manager" Jul 29 20:58:13.740121 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.740123 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="05eed357-d4ae-409a-ac02-2830e53d7099" containerName="manager" Jul 29 20:58:13.740349 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.740229 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="05eed357-d4ae-409a-ac02-2830e53d7099" containerName="manager" Jul 29 20:58:13.740992 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.740958 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:58:13.743242 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.743224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" Jul 29 20:58:13.748186 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.748162 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ctkg9\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="05eed357-d4ae-409a-ac02-2830e53d7099" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:13.748416 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.748396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnw4g\" (UniqueName: \"kubernetes.io/projected/8103c110-825d-4713-963b-8ffb82e03cd8-kube-api-access-wnw4g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r87tg\" (UID: \"8103c110-825d-4713-963b-8ffb82e03cd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" Jul 29 20:58:13.748478 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.748462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72024507-e8e3-479d-9add-3a9bb6c2f93c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9hchq\" (UID: \"72024507-e8e3-479d-9add-3a9bb6c2f93c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:13.748532 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.748496 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8103c110-825d-4713-963b-8ffb82e03cd8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r87tg\" (UID: \"8103c110-825d-4713-963b-8ffb82e03cd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" Jul 29 20:58:13.748532 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.748526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzr5k\" (UniqueName: \"kubernetes.io/projected/72024507-e8e3-479d-9add-3a9bb6c2f93c-kube-api-access-fzr5k\") pod \"kuadrant-operator-controller-manager-86878c769f-9hchq\" (UID: \"72024507-e8e3-479d-9add-3a9bb6c2f93c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:13.748819 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.748803 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72024507-e8e3-479d-9add-3a9bb6c2f93c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9hchq\" (UID: \"72024507-e8e3-479d-9add-3a9bb6c2f93c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:13.750723 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.750701 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:58:13.755132 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.755107 2582 projected.go:196] Error preparing data for projected volume kube-api-access-fzr5k for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-9hchq" not found Jul 29 20:58:13.755220 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.755167 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72024507-e8e3-479d-9add-3a9bb6c2f93c-kube-api-access-fzr5k podName:72024507-e8e3-479d-9add-3a9bb6c2f93c nodeName:}" failed. No retries permitted until 2026-07-29 20:58:14.255152069 +0000 UTC m=+642.378741892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fzr5k" (UniqueName: "kubernetes.io/projected/72024507-e8e3-479d-9add-3a9bb6c2f93c-kube-api-access-fzr5k") pod "kuadrant-operator-controller-manager-86878c769f-9hchq" (UID: "72024507-e8e3-479d-9add-3a9bb6c2f93c") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-9hchq" not found Jul 29 20:58:13.764953 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.764929 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx"] Jul 29 20:58:13.849307 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.849274 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6hn\" (UniqueName: \"kubernetes.io/projected/a1ceae93-dc20-4cd6-b15d-aba70fc993dc-kube-api-access-fq6hn\") pod \"kuadrant-operator-controller-manager-86878c769f-m57rx\" (UID: \"a1ceae93-dc20-4cd6-b15d-aba70fc993dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" Jul 29 20:58:13.849512 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.849323 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnw4g\" (UniqueName: \"kubernetes.io/projected/8103c110-825d-4713-963b-8ffb82e03cd8-kube-api-access-wnw4g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r87tg\" (UID: \"8103c110-825d-4713-963b-8ffb82e03cd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" Jul 29 20:58:13.849512 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.849489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8103c110-825d-4713-963b-8ffb82e03cd8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r87tg\" (UID: \"8103c110-825d-4713-963b-8ffb82e03cd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" Jul 29 20:58:13.849629 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.849585 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1ceae93-dc20-4cd6-b15d-aba70fc993dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-m57rx\" (UID: \"a1ceae93-dc20-4cd6-b15d-aba70fc993dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" Jul 29 20:58:13.849877 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.849851 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8103c110-825d-4713-963b-8ffb82e03cd8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r87tg\" (UID: \"8103c110-825d-4713-963b-8ffb82e03cd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" Jul 29 20:58:13.863078 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.863005 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnw4g\" (UniqueName: \"kubernetes.io/projected/8103c110-825d-4713-963b-8ffb82e03cd8-kube-api-access-wnw4g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r87tg\" (UID: \"8103c110-825d-4713-963b-8ffb82e03cd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" Jul 29 20:58:13.911916 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.911883 2582 generic.go:356] "Generic (PLEG): container finished" podID="05eed357-d4ae-409a-ac02-2830e53d7099" containerID="13e35796c47e6f3095f96ffa7c5a66512d0fc2c9f24bee3e711e211a57b1ba47" exitCode=0 Jul 29 20:58:13.913833 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.913808 2582 generic.go:356] "Generic (PLEG): container finished" podID="efa0e58f-36fa-4038-acf8-6dc90e493350" containerID="13ac81858bf8c8768c867f3bd40488a960c93112518ee3356c4b50c21edfe9cd" exitCode=0 Jul 29 20:58:13.913951 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.913925 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.914101 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.914075 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:13.916943 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.916922 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ctkg9\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="05eed357-d4ae-409a-ac02-2830e53d7099" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:13.919286 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.919256 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:58:13.921126 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.921108 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.921589 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.921568 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-c6f8q\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.923712 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.923690 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:58:13.924381 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.924363 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:13.925791 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.925769 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-c6f8q\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.927899 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.927873 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ctkg9\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="05eed357-d4ae-409a-ac02-2830e53d7099" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:13.930152 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.930126 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-9hchq\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="72024507-e8e3-479d-9add-3a9bb6c2f93c" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:13.932333 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.932305 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ctkg9\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="05eed357-d4ae-409a-ac02-2830e53d7099" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:13.934549 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.934427 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-9hchq\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="72024507-e8e3-479d-9add-3a9bb6c2f93c" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:13.936512 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.936489 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:58:13.938759 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.938733 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-c6f8q\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.942904 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.942883 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:58:13.945692 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.945638 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-9hchq\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="72024507-e8e3-479d-9add-3a9bb6c2f93c" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:13.947617 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.947595 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:58:13.949468 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.949441 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-c6f8q\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.950750 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.950730 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72024507-e8e3-479d-9add-3a9bb6c2f93c-extensions-socket-volume\") pod \"72024507-e8e3-479d-9add-3a9bb6c2f93c\" (UID: \"72024507-e8e3-479d-9add-3a9bb6c2f93c\") " Jul 29 20:58:13.950835 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.950779 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934-extensions-socket-volume\") pod \"0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934\" (UID: \"0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934\") " Jul 29 20:58:13.950902 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.950888 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6zxq\" (UniqueName: \"kubernetes.io/projected/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934-kube-api-access-b6zxq\") pod \"0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934\" (UID: \"0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934\") " Jul 29 20:58:13.951076 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.951004 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72024507-e8e3-479d-9add-3a9bb6c2f93c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "72024507-e8e3-479d-9add-3a9bb6c2f93c" (UID: "72024507-e8e3-479d-9add-3a9bb6c2f93c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:58:13.951166 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.951037 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934" (UID: "0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:58:13.951166 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.951038 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6hn\" (UniqueName: \"kubernetes.io/projected/a1ceae93-dc20-4cd6-b15d-aba70fc993dc-kube-api-access-fq6hn\") pod \"kuadrant-operator-controller-manager-86878c769f-m57rx\" (UID: \"a1ceae93-dc20-4cd6-b15d-aba70fc993dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" Jul 29 20:58:13.951299 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.951181 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1ceae93-dc20-4cd6-b15d-aba70fc993dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-m57rx\" (UID: \"a1ceae93-dc20-4cd6-b15d-aba70fc993dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" Jul 29 20:58:13.951299 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.951255 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzr5k\" (UniqueName: \"kubernetes.io/projected/72024507-e8e3-479d-9add-3a9bb6c2f93c-kube-api-access-fzr5k\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:58:13.951299 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.951271 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72024507-e8e3-479d-9add-3a9bb6c2f93c-extensions-socket-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:58:13.951299 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.951292 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934-extensions-socket-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:58:13.951491 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.951379 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ctkg9\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="05eed357-d4ae-409a-ac02-2830e53d7099" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:13.951562 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.951546 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1ceae93-dc20-4cd6-b15d-aba70fc993dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-m57rx\" (UID: \"a1ceae93-dc20-4cd6-b15d-aba70fc993dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" Jul 29 20:58:13.951667 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.951652 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:13.953195 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.953176 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934-kube-api-access-b6zxq" (OuterVolumeSpecName: "kube-api-access-b6zxq") pod "0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934" (UID: "0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934"). InnerVolumeSpecName "kube-api-access-b6zxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:58:13.953633 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.953616 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-c6f8q\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:13.955589 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.955572 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ctkg9\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="05eed357-d4ae-409a-ac02-2830e53d7099" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:13.957370 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.957349 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-9hchq\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="72024507-e8e3-479d-9add-3a9bb6c2f93c" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:13.959294 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:13.959276 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6hn\" (UniqueName: \"kubernetes.io/projected/a1ceae93-dc20-4cd6-b15d-aba70fc993dc-kube-api-access-fq6hn\") pod \"kuadrant-operator-controller-manager-86878c769f-m57rx\" (UID: \"a1ceae93-dc20-4cd6-b15d-aba70fc993dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" Jul 29 20:58:13.959446 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:13.959429 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8776s\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:58:14.051885 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.051860 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnm8l\" (UniqueName: \"kubernetes.io/projected/05eed357-d4ae-409a-ac02-2830e53d7099-kube-api-access-wnm8l\") pod \"05eed357-d4ae-409a-ac02-2830e53d7099\" (UID: \"05eed357-d4ae-409a-ac02-2830e53d7099\") " Jul 29 20:58:14.052019 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.051900 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvcxz\" (UniqueName: \"kubernetes.io/projected/efa0e58f-36fa-4038-acf8-6dc90e493350-kube-api-access-bvcxz\") pod \"efa0e58f-36fa-4038-acf8-6dc90e493350\" (UID: \"efa0e58f-36fa-4038-acf8-6dc90e493350\") " Jul 29 20:58:14.052019 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.051941 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/efa0e58f-36fa-4038-acf8-6dc90e493350-extensions-socket-volume\") pod \"efa0e58f-36fa-4038-acf8-6dc90e493350\" (UID: \"efa0e58f-36fa-4038-acf8-6dc90e493350\") " Jul 29 20:58:14.052142 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.052129 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6zxq\" (UniqueName: \"kubernetes.io/projected/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934-kube-api-access-b6zxq\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:58:14.052523 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.052497 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa0e58f-36fa-4038-acf8-6dc90e493350-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "efa0e58f-36fa-4038-acf8-6dc90e493350" (UID: "efa0e58f-36fa-4038-acf8-6dc90e493350"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:58:14.054013 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.053993 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05eed357-d4ae-409a-ac02-2830e53d7099-kube-api-access-wnm8l" (OuterVolumeSpecName: "kube-api-access-wnm8l") pod "05eed357-d4ae-409a-ac02-2830e53d7099" (UID: "05eed357-d4ae-409a-ac02-2830e53d7099"). InnerVolumeSpecName "kube-api-access-wnm8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:58:14.054110 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.054019 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa0e58f-36fa-4038-acf8-6dc90e493350-kube-api-access-bvcxz" (OuterVolumeSpecName: "kube-api-access-bvcxz") pod "efa0e58f-36fa-4038-acf8-6dc90e493350" (UID: "efa0e58f-36fa-4038-acf8-6dc90e493350"). InnerVolumeSpecName "kube-api-access-bvcxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:58:14.097962 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.097942 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" Jul 29 20:58:14.103881 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.103862 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" Jul 29 20:58:14.154798 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.153540 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnm8l\" (UniqueName: \"kubernetes.io/projected/05eed357-d4ae-409a-ac02-2830e53d7099-kube-api-access-wnm8l\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:58:14.154798 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.153572 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvcxz\" (UniqueName: \"kubernetes.io/projected/efa0e58f-36fa-4038-acf8-6dc90e493350-kube-api-access-bvcxz\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:58:14.154798 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.153597 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/efa0e58f-36fa-4038-acf8-6dc90e493350-extensions-socket-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:58:14.249662 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.249638 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg"] Jul 29 20:58:14.251297 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:58:14.251267 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8103c110_825d_4713_963b_8ffb82e03cd8.slice/crio-599266f264b1b70ef1f71f1ccc212e121b8c7a8c9d68a8270149c14aae27712d WatchSource:0}: Error finding container 599266f264b1b70ef1f71f1ccc212e121b8c7a8c9d68a8270149c14aae27712d: Status 404 returned error can't find the container with id 599266f264b1b70ef1f71f1ccc212e121b8c7a8c9d68a8270149c14aae27712d Jul 29 20:58:14.264107 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.264081 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx"] Jul 29 20:58:14.265766 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:58:14.265743 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ceae93_dc20_4cd6_b15d_aba70fc993dc.slice/crio-2861e22a6c40e11ccae15fd6866ecb52d642ec8a3dd3b54abcba25e11912ec80 WatchSource:0}: Error finding container 2861e22a6c40e11ccae15fd6866ecb52d642ec8a3dd3b54abcba25e11912ec80: Status 404 returned error can't find the container with id 2861e22a6c40e11ccae15fd6866ecb52d642ec8a3dd3b54abcba25e11912ec80 Jul 29 20:58:14.493088 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.493058 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05eed357-d4ae-409a-ac02-2830e53d7099" path="/var/lib/kubelet/pods/05eed357-d4ae-409a-ac02-2830e53d7099/volumes" Jul 29 20:58:14.493449 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.493408 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934" path="/var/lib/kubelet/pods/0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934/volumes" Jul 29 20:58:14.493595 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.493584 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72024507-e8e3-479d-9add-3a9bb6c2f93c" path="/var/lib/kubelet/pods/72024507-e8e3-479d-9add-3a9bb6c2f93c/volumes" Jul 29 20:58:14.493798 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.493788 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efa0e58f-36fa-4038-acf8-6dc90e493350" path="/var/lib/kubelet/pods/efa0e58f-36fa-4038-acf8-6dc90e493350/volumes" Jul 29 20:58:14.921099 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.920987 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" event={"ID":"a1ceae93-dc20-4cd6-b15d-aba70fc993dc","Type":"ContainerStarted","Data":"b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c"} Jul 29 20:58:14.921099 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.921078 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" event={"ID":"a1ceae93-dc20-4cd6-b15d-aba70fc993dc","Type":"ContainerStarted","Data":"2861e22a6c40e11ccae15fd6866ecb52d642ec8a3dd3b54abcba25e11912ec80"} Jul 29 20:58:14.921315 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.921112 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" Jul 29 20:58:14.922311 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.922289 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctkg9" Jul 29 20:58:14.922436 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.922349 2582 scope.go:117] "RemoveContainer" containerID="13e35796c47e6f3095f96ffa7c5a66512d0fc2c9f24bee3e711e211a57b1ba47" Jul 29 20:58:14.924195 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.924166 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8776s" Jul 29 20:58:14.925936 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.925910 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" event={"ID":"8103c110-825d-4713-963b-8ffb82e03cd8","Type":"ContainerStarted","Data":"5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68"} Jul 29 20:58:14.926058 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.925942 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:14.926058 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.925957 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:14.926058 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.925962 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" event={"ID":"8103c110-825d-4713-963b-8ffb82e03cd8","Type":"ContainerStarted","Data":"599266f264b1b70ef1f71f1ccc212e121b8c7a8c9d68a8270149c14aae27712d"} Jul 29 20:58:14.932954 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.932934 2582 scope.go:117] "RemoveContainer" containerID="13ac81858bf8c8768c867f3bd40488a960c93112518ee3356c4b50c21edfe9cd" Jul 29 20:58:14.958422 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.958374 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" podStartSLOduration=1.958360315 podStartE2EDuration="1.958360315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:58:14.956016859 +0000 UTC m=+643.079606702" watchObservedRunningTime="2026-07-29 20:58:14.958360315 +0000 UTC m=+643.081950217" Jul 29 20:58:14.983151 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:14.983108 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" podStartSLOduration=1.9830948309999998 podStartE2EDuration="1.983094831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:58:14.981487876 +0000 UTC m=+643.105077709" watchObservedRunningTime="2026-07-29 20:58:14.983094831 +0000 UTC m=+643.106684672" Jul 29 20:58:14.983656 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:14.983631 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-9hchq\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="72024507-e8e3-479d-9add-3a9bb6c2f93c" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hchq" Jul 29 20:58:14.985471 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:14.985452 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-c6f8q\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="0c99dcbf-8bec-4eb7-a61f-2d2ef4e55934" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c6f8q" Jul 29 20:58:15.932431 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:15.932402 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" Jul 29 20:58:25.934840 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:25.934807 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" Jul 29 20:58:25.991513 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:25.991479 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg"] Jul 29 20:58:25.991721 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:25.991660 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" podUID="8103c110-825d-4713-963b-8ffb82e03cd8" containerName="manager" containerID="cri-o://5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68" gracePeriod=10 Jul 29 20:58:25.993959 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:25.993935 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" Jul 29 20:58:26.235007 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.234980 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" Jul 29 20:58:26.353556 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.353526 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnw4g\" (UniqueName: \"kubernetes.io/projected/8103c110-825d-4713-963b-8ffb82e03cd8-kube-api-access-wnw4g\") pod \"8103c110-825d-4713-963b-8ffb82e03cd8\" (UID: \"8103c110-825d-4713-963b-8ffb82e03cd8\") " Jul 29 20:58:26.353705 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.353582 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8103c110-825d-4713-963b-8ffb82e03cd8-extensions-socket-volume\") pod \"8103c110-825d-4713-963b-8ffb82e03cd8\" (UID: \"8103c110-825d-4713-963b-8ffb82e03cd8\") " Jul 29 20:58:26.354002 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.353972 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8103c110-825d-4713-963b-8ffb82e03cd8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8103c110-825d-4713-963b-8ffb82e03cd8" (UID: "8103c110-825d-4713-963b-8ffb82e03cd8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:58:26.355691 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.355674 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8103c110-825d-4713-963b-8ffb82e03cd8-kube-api-access-wnw4g" (OuterVolumeSpecName: "kube-api-access-wnw4g") pod "8103c110-825d-4713-963b-8ffb82e03cd8" (UID: "8103c110-825d-4713-963b-8ffb82e03cd8"). InnerVolumeSpecName "kube-api-access-wnw4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:58:26.454994 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.454930 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnw4g\" (UniqueName: \"kubernetes.io/projected/8103c110-825d-4713-963b-8ffb82e03cd8-kube-api-access-wnw4g\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:58:26.454994 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.454953 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8103c110-825d-4713-963b-8ffb82e03cd8-extensions-socket-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:58:26.985798 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.985764 2582 generic.go:356] "Generic (PLEG): container finished" podID="8103c110-825d-4713-963b-8ffb82e03cd8" containerID="5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68" exitCode=0 Jul 29 20:58:26.986299 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.985820 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" Jul 29 20:58:26.986299 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.985862 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" event={"ID":"8103c110-825d-4713-963b-8ffb82e03cd8","Type":"ContainerDied","Data":"5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68"} Jul 29 20:58:26.986299 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.985926 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg" event={"ID":"8103c110-825d-4713-963b-8ffb82e03cd8","Type":"ContainerDied","Data":"599266f264b1b70ef1f71f1ccc212e121b8c7a8c9d68a8270149c14aae27712d"} Jul 29 20:58:26.986299 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.985962 2582 scope.go:117] "RemoveContainer" containerID="5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68" Jul 29 20:58:26.994902 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.994702 2582 scope.go:117] "RemoveContainer" containerID="5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68" Jul 29 20:58:26.994994 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:26.994976 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68\": container with ID starting with 5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68 not found: ID does not exist" containerID="5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68" Jul 29 20:58:26.995056 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:26.995002 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68"} err="failed to get container status \"5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68\": rpc error: code = NotFound desc = could not find container \"5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68\": container with ID starting with 5d3bd66ecbcd0e0c122d9a504916735687115ccd35b04af200c346912d860d68 not found: ID does not exist" Jul 29 20:58:27.005406 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:27.005385 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg"] Jul 29 20:58:27.011305 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:27.011286 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r87tg"] Jul 29 20:58:28.493142 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:28.493110 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8103c110-825d-4713-963b-8ffb82e03cd8" path="/var/lib/kubelet/pods/8103c110-825d-4713-963b-8ffb82e03cd8/volumes" Jul 29 20:58:42.294229 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.294198 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs"] Jul 29 20:58:42.294641 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.294624 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8103c110-825d-4713-963b-8ffb82e03cd8" containerName="manager" Jul 29 20:58:42.294641 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.294642 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8103c110-825d-4713-963b-8ffb82e03cd8" containerName="manager" Jul 29 20:58:42.294725 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.294704 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8103c110-825d-4713-963b-8ffb82e03cd8" containerName="manager" Jul 29 20:58:42.300877 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.300858 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.303296 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.303279 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2kg6x\"" Jul 29 20:58:42.311803 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.311777 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs"] Jul 29 20:58:42.388699 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.388664 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.388849 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.388730 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.388849 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.388776 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.388849 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.388814 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.388849 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.388831 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.389012 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.388918 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hncvw\" (UniqueName: \"kubernetes.io/projected/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-kube-api-access-hncvw\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.389012 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.388993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.389116 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.389056 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.389157 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.389130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.489570 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.489522 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.489570 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.489566 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.489811 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.489609 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.489811 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.489636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.489811 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.489675 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hncvw\" (UniqueName: \"kubernetes.io/projected/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-kube-api-access-hncvw\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.489811 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.489719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.489811 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.489748 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.490118 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.489811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.490118 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.489853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.490118 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.489904 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.490312 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.490289 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.490678 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.490314 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.490809 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.490520 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.491168 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.491137 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.492606 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.492574 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.492707 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.492617 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.497795 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.497771 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.497999 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.497979 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hncvw\" (UniqueName: \"kubernetes.io/projected/07571bcc-6ebe-4a1b-8749-f072d9e5f6a9-kube-api-access-hncvw\") pod \"maas-default-gateway-openshift-default-78bb78ff78-qbhbs\" (UID: \"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.613270 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.613196 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:42.749146 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.749124 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs"] Jul 29 20:58:42.750945 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:58:42.750918 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07571bcc_6ebe_4a1b_8749_f072d9e5f6a9.slice/crio-9d262f4183c38359d18e5cc82f221ce88deb80c75a2166e98c96a0fff552ae0e WatchSource:0}: Error finding container 9d262f4183c38359d18e5cc82f221ce88deb80c75a2166e98c96a0fff552ae0e: Status 404 returned error can't find the container with id 9d262f4183c38359d18e5cc82f221ce88deb80c75a2166e98c96a0fff552ae0e Jul 29 20:58:42.753073 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.753039 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 20:58:42.753191 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.753112 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 20:58:42.753191 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:42.753158 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 20:58:43.063670 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:43.063634 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" event={"ID":"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9","Type":"ContainerStarted","Data":"5b818452d128e28541d8e66f836ea3624764e9235d92a8ddab8bf7a44b64d4e1"} Jul 29 20:58:43.063842 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:43.063686 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" event={"ID":"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9","Type":"ContainerStarted","Data":"9d262f4183c38359d18e5cc82f221ce88deb80c75a2166e98c96a0fff552ae0e"} Jul 29 20:58:43.083698 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:43.083644 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" podStartSLOduration=1.083629044 podStartE2EDuration="1.083629044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:58:43.081927461 +0000 UTC m=+671.205517328" watchObservedRunningTime="2026-07-29 20:58:43.083629044 +0000 UTC m=+671.207218885" Jul 29 20:58:43.613598 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:43.613568 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:43.618297 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:43.618275 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:44.068163 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:44.068138 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:44.069070 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:44.069049 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 20:58:44.846041 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:44.845996 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-bh8dw"] Jul 29 20:58:44.848462 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:44.848441 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bh8dw" Jul 29 20:58:44.850739 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:44.850722 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5cvzg\"" Jul 29 20:58:44.854246 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:44.854227 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-bh8dw"] Jul 29 20:58:45.006120 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.006087 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-dggcf"] Jul 29 20:58:45.008556 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.008535 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dggcf" Jul 29 20:58:45.013733 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.013706 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r8v2\" (UniqueName: \"kubernetes.io/projected/b6cd611d-ef0e-447e-8db5-9bfcebaad0c6-kube-api-access-6r8v2\") pod \"authorino-58cd955675-bh8dw\" (UID: \"b6cd611d-ef0e-447e-8db5-9bfcebaad0c6\") " pod="kuadrant-system/authorino-58cd955675-bh8dw" Jul 29 20:58:45.016831 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.016810 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dggcf"] Jul 29 20:58:45.115127 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.115056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r8v2\" (UniqueName: \"kubernetes.io/projected/b6cd611d-ef0e-447e-8db5-9bfcebaad0c6-kube-api-access-6r8v2\") pod \"authorino-58cd955675-bh8dw\" (UID: \"b6cd611d-ef0e-447e-8db5-9bfcebaad0c6\") " pod="kuadrant-system/authorino-58cd955675-bh8dw" Jul 29 20:58:45.115271 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.115131 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mfzv\" (UniqueName: \"kubernetes.io/projected/61cc5ea7-861b-4782-a5af-d4c31d529a22-kube-api-access-2mfzv\") pod \"authorino-75d556bd77-dggcf\" (UID: \"61cc5ea7-861b-4782-a5af-d4c31d529a22\") " pod="kuadrant-system/authorino-75d556bd77-dggcf" Jul 29 20:58:45.122581 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.122559 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r8v2\" (UniqueName: \"kubernetes.io/projected/b6cd611d-ef0e-447e-8db5-9bfcebaad0c6-kube-api-access-6r8v2\") pod \"authorino-58cd955675-bh8dw\" (UID: \"b6cd611d-ef0e-447e-8db5-9bfcebaad0c6\") " pod="kuadrant-system/authorino-58cd955675-bh8dw" Jul 29 20:58:45.158247 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.158220 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bh8dw" Jul 29 20:58:45.215986 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.215957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mfzv\" (UniqueName: \"kubernetes.io/projected/61cc5ea7-861b-4782-a5af-d4c31d529a22-kube-api-access-2mfzv\") pod \"authorino-75d556bd77-dggcf\" (UID: \"61cc5ea7-861b-4782-a5af-d4c31d529a22\") " pod="kuadrant-system/authorino-75d556bd77-dggcf" Jul 29 20:58:45.224908 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.224881 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mfzv\" (UniqueName: \"kubernetes.io/projected/61cc5ea7-861b-4782-a5af-d4c31d529a22-kube-api-access-2mfzv\") pod \"authorino-75d556bd77-dggcf\" (UID: \"61cc5ea7-861b-4782-a5af-d4c31d529a22\") " pod="kuadrant-system/authorino-75d556bd77-dggcf" Jul 29 20:58:45.288345 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.288314 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-bh8dw"] Jul 29 20:58:45.290243 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:58:45.290215 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6cd611d_ef0e_447e_8db5_9bfcebaad0c6.slice/crio-f66c4756b8aa03f18d9351f48a1dd602fa06f60106fa161e29ada8e1e0e0ebb0 WatchSource:0}: Error finding container f66c4756b8aa03f18d9351f48a1dd602fa06f60106fa161e29ada8e1e0e0ebb0: Status 404 returned error can't find the container with id f66c4756b8aa03f18d9351f48a1dd602fa06f60106fa161e29ada8e1e0e0ebb0 Jul 29 20:58:45.317637 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.317612 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dggcf" Jul 29 20:58:45.446894 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:45.446859 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dggcf"] Jul 29 20:58:45.448955 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:58:45.448925 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61cc5ea7_861b_4782_a5af_d4c31d529a22.slice/crio-ec60c93c93d4be87524728b4e17c9458dc905a2835b8a10ee0c56ad56d4b96ff WatchSource:0}: Error finding container ec60c93c93d4be87524728b4e17c9458dc905a2835b8a10ee0c56ad56d4b96ff: Status 404 returned error can't find the container with id ec60c93c93d4be87524728b4e17c9458dc905a2835b8a10ee0c56ad56d4b96ff Jul 29 20:58:46.050776 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.050743 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 20:58:46.054202 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.054181 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" Jul 29 20:58:46.062148 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.062123 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 29 20:58:46.062388 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.062372 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hlkjq\"" Jul 29 20:58:46.070882 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.070853 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 20:58:46.078816 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.078136 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 20:58:46.086658 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.086608 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bh8dw" event={"ID":"b6cd611d-ef0e-447e-8db5-9bfcebaad0c6","Type":"ContainerStarted","Data":"f66c4756b8aa03f18d9351f48a1dd602fa06f60106fa161e29ada8e1e0e0ebb0"} Jul 29 20:58:46.091382 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.091345 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dggcf" event={"ID":"61cc5ea7-861b-4782-a5af-d4c31d529a22","Type":"ContainerStarted","Data":"ec60c93c93d4be87524728b4e17c9458dc905a2835b8a10ee0c56ad56d4b96ff"} Jul 29 20:58:46.227487 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.226851 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a24582f-27fe-4c7f-8ee3-092b06e8de12-config-file\") pod \"limitador-limitador-54d6d988d5-79tv9\" (UID: \"9a24582f-27fe-4c7f-8ee3-092b06e8de12\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" Jul 29 20:58:46.227487 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.226980 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdbz\" (UniqueName: \"kubernetes.io/projected/9a24582f-27fe-4c7f-8ee3-092b06e8de12-kube-api-access-skdbz\") pod \"limitador-limitador-54d6d988d5-79tv9\" (UID: \"9a24582f-27fe-4c7f-8ee3-092b06e8de12\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" Jul 29 20:58:46.327731 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.327648 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a24582f-27fe-4c7f-8ee3-092b06e8de12-config-file\") pod \"limitador-limitador-54d6d988d5-79tv9\" (UID: \"9a24582f-27fe-4c7f-8ee3-092b06e8de12\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" Jul 29 20:58:46.327915 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.327746 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdbz\" (UniqueName: \"kubernetes.io/projected/9a24582f-27fe-4c7f-8ee3-092b06e8de12-kube-api-access-skdbz\") pod \"limitador-limitador-54d6d988d5-79tv9\" (UID: \"9a24582f-27fe-4c7f-8ee3-092b06e8de12\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" Jul 29 20:58:46.328908 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.328869 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a24582f-27fe-4c7f-8ee3-092b06e8de12-config-file\") pod \"limitador-limitador-54d6d988d5-79tv9\" (UID: \"9a24582f-27fe-4c7f-8ee3-092b06e8de12\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" Jul 29 20:58:46.339190 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.339141 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdbz\" (UniqueName: \"kubernetes.io/projected/9a24582f-27fe-4c7f-8ee3-092b06e8de12-kube-api-access-skdbz\") pod \"limitador-limitador-54d6d988d5-79tv9\" (UID: \"9a24582f-27fe-4c7f-8ee3-092b06e8de12\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" Jul 29 20:58:46.384281 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.383996 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" Jul 29 20:58:46.587249 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:46.587220 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 20:58:46.603330 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:58:46.603301 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a24582f_27fe_4c7f_8ee3_092b06e8de12.slice/crio-58504bb22175cd8cc67b60713bf5ea156d9ec4cc8fb36111f24a413e02acafce WatchSource:0}: Error finding container 58504bb22175cd8cc67b60713bf5ea156d9ec4cc8fb36111f24a413e02acafce: Status 404 returned error can't find the container with id 58504bb22175cd8cc67b60713bf5ea156d9ec4cc8fb36111f24a413e02acafce Jul 29 20:58:47.099224 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:47.099183 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" event={"ID":"9a24582f-27fe-4c7f-8ee3-092b06e8de12","Type":"ContainerStarted","Data":"58504bb22175cd8cc67b60713bf5ea156d9ec4cc8fb36111f24a413e02acafce"} Jul 29 20:58:49.115321 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:49.115287 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bh8dw" event={"ID":"b6cd611d-ef0e-447e-8db5-9bfcebaad0c6","Type":"ContainerStarted","Data":"938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4"} Jul 29 20:58:49.116967 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:49.116926 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dggcf" event={"ID":"61cc5ea7-861b-4782-a5af-d4c31d529a22","Type":"ContainerStarted","Data":"e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6"} Jul 29 20:58:49.131293 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:49.131245 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-bh8dw" podStartSLOduration=1.697074791 podStartE2EDuration="5.131231128s" totalImagesPullingTime="3.434156337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:58:49.130271816 +0000 UTC m=+677.253861652" watchObservedRunningTime="2026-07-29 20:58:49.131231128 +0000 UTC m=+677.254820969" Jul 29 20:58:49.147140 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:49.147093 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-dggcf" podStartSLOduration=1.8584750749999999 podStartE2EDuration="5.147081279s" totalImagesPullingTime="3.288606204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:58:49.145743545 +0000 UTC m=+677.269333392" watchObservedRunningTime="2026-07-29 20:58:49.147081279 +0000 UTC m=+677.270671118" Jul 29 20:58:49.175690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:49.175658 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-bh8dw"] Jul 29 20:58:51.129180 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:51.129148 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" event={"ID":"9a24582f-27fe-4c7f-8ee3-092b06e8de12","Type":"ContainerStarted","Data":"e7c4f4efd8ef0598696116c385a10317fbd9951ddbd093fc6284f7c3bb4508fa"} Jul 29 20:58:51.129606 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:51.129199 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-bh8dw" podUID="b6cd611d-ef0e-447e-8db5-9bfcebaad0c6" containerName="authorino" containerID="cri-o://938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4" gracePeriod=30 Jul 29 20:58:51.129606 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:51.129407 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" Jul 29 20:58:51.162333 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:51.162281 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" podStartSLOduration=1.449940561 podStartE2EDuration="5.162263602s" totalImagesPullingTime="3.712323041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:58:51.161048607 +0000 UTC m=+679.284638440" watchObservedRunningTime="2026-07-29 20:58:51.162263602 +0000 UTC m=+679.285853443" Jul 29 20:58:51.377057 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:51.377018 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bh8dw" Jul 29 20:58:51.474908 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:51.474822 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r8v2\" (UniqueName: \"kubernetes.io/projected/b6cd611d-ef0e-447e-8db5-9bfcebaad0c6-kube-api-access-6r8v2\") pod \"b6cd611d-ef0e-447e-8db5-9bfcebaad0c6\" (UID: \"b6cd611d-ef0e-447e-8db5-9bfcebaad0c6\") " Jul 29 20:58:51.477057 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:51.477012 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cd611d-ef0e-447e-8db5-9bfcebaad0c6-kube-api-access-6r8v2" (OuterVolumeSpecName: "kube-api-access-6r8v2") pod "b6cd611d-ef0e-447e-8db5-9bfcebaad0c6" (UID: "b6cd611d-ef0e-447e-8db5-9bfcebaad0c6"). InnerVolumeSpecName "kube-api-access-6r8v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:58:51.575974 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:51.575943 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r8v2\" (UniqueName: \"kubernetes.io/projected/b6cd611d-ef0e-447e-8db5-9bfcebaad0c6-kube-api-access-6r8v2\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:58:52.135430 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:52.135401 2582 generic.go:356] "Generic (PLEG): container finished" podID="b6cd611d-ef0e-447e-8db5-9bfcebaad0c6" containerID="938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4" exitCode=0 Jul 29 20:58:52.135836 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:52.135444 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bh8dw" Jul 29 20:58:52.135836 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:52.135493 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bh8dw" event={"ID":"b6cd611d-ef0e-447e-8db5-9bfcebaad0c6","Type":"ContainerDied","Data":"938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4"} Jul 29 20:58:52.135836 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:52.135533 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bh8dw" event={"ID":"b6cd611d-ef0e-447e-8db5-9bfcebaad0c6","Type":"ContainerDied","Data":"f66c4756b8aa03f18d9351f48a1dd602fa06f60106fa161e29ada8e1e0e0ebb0"} Jul 29 20:58:52.135836 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:52.135552 2582 scope.go:117] "RemoveContainer" containerID="938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4" Jul 29 20:58:52.144558 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:52.144541 2582 scope.go:117] "RemoveContainer" containerID="938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4" Jul 29 20:58:52.144812 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:58:52.144796 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4\": container with ID starting with 938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4 not found: ID does not exist" containerID="938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4" Jul 29 20:58:52.144860 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:52.144821 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4"} err="failed to get container status \"938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4\": rpc error: code = NotFound desc = could not find container \"938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4\": container with ID starting with 938485f707d8cc90f61857664fcd4227d3d0e533549ce3f88052d900dd108eb4 not found: ID does not exist" Jul 29 20:58:52.156367 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:52.156340 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-bh8dw"] Jul 29 20:58:52.158690 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:52.158671 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-bh8dw"] Jul 29 20:58:52.492813 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:58:52.492784 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6cd611d-ef0e-447e-8db5-9bfcebaad0c6" path="/var/lib/kubelet/pods/b6cd611d-ef0e-447e-8db5-9bfcebaad0c6/volumes" Jul 29 20:59:02.136362 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:02.136336 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-79tv9" Jul 29 20:59:17.601234 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.601203 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ctpwj"] Jul 29 20:59:17.601590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.601530 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cd611d-ef0e-447e-8db5-9bfcebaad0c6" containerName="authorino" Jul 29 20:59:17.601590 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.601540 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cd611d-ef0e-447e-8db5-9bfcebaad0c6" containerName="authorino" Jul 29 20:59:17.601667 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.601613 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cd611d-ef0e-447e-8db5-9bfcebaad0c6" containerName="authorino" Jul 29 20:59:17.604652 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.604632 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ctpwj" Jul 29 20:59:17.615189 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.615163 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ctpwj"] Jul 29 20:59:17.686963 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.686933 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76d7f\" (UniqueName: \"kubernetes.io/projected/f5709b9a-b593-4817-b254-78733fc5dbce-kube-api-access-76d7f\") pod \"authorino-5d4b67c446-ctpwj\" (UID: \"f5709b9a-b593-4817-b254-78733fc5dbce\") " pod="kuadrant-system/authorino-5d4b67c446-ctpwj" Jul 29 20:59:17.787531 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.787493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76d7f\" (UniqueName: \"kubernetes.io/projected/f5709b9a-b593-4817-b254-78733fc5dbce-kube-api-access-76d7f\") pod \"authorino-5d4b67c446-ctpwj\" (UID: \"f5709b9a-b593-4817-b254-78733fc5dbce\") " pod="kuadrant-system/authorino-5d4b67c446-ctpwj" Jul 29 20:59:17.796098 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.796080 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76d7f\" (UniqueName: \"kubernetes.io/projected/f5709b9a-b593-4817-b254-78733fc5dbce-kube-api-access-76d7f\") pod \"authorino-5d4b67c446-ctpwj\" (UID: \"f5709b9a-b593-4817-b254-78733fc5dbce\") " pod="kuadrant-system/authorino-5d4b67c446-ctpwj" Jul 29 20:59:17.835865 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.835834 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ctpwj"] Jul 29 20:59:17.836083 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.836071 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ctpwj" Jul 29 20:59:17.864343 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.864282 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58b6dd77c5-pm22m"] Jul 29 20:59:17.868515 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.868498 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b6dd77c5-pm22m" Jul 29 20:59:17.874614 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.874565 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58b6dd77c5-pm22m"] Jul 29 20:59:17.918387 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.918357 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58b6dd77c5-pm22m"] Jul 29 20:59:17.918638 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:59:17.918620 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6b494], unattached volumes=[], failed to process volumes=[kube-api-access-6b494]: context canceled" pod="kuadrant-system/authorino-58b6dd77c5-pm22m" podUID="5aa9de95-4785-4ec9-a9ab-be931e6ee65f" Jul 29 20:59:17.948437 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.948409 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59f9b5458c-7mg4b"] Jul 29 20:59:17.951712 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.951691 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59f9b5458c-7mg4b" Jul 29 20:59:17.954210 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.954183 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 29 20:59:17.959412 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.959387 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59f9b5458c-7mg4b"] Jul 29 20:59:17.977182 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.977163 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ctpwj"] Jul 29 20:59:17.979161 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:59:17.979138 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5709b9a_b593_4817_b254_78733fc5dbce.slice/crio-dd72a4645b6c4cb9123bda5b07f787ee4468d0f5118f82ff5ed81bf650701e82 WatchSource:0}: Error finding container dd72a4645b6c4cb9123bda5b07f787ee4468d0f5118f82ff5ed81bf650701e82: Status 404 returned error can't find the container with id dd72a4645b6c4cb9123bda5b07f787ee4468d0f5118f82ff5ed81bf650701e82 Jul 29 20:59:17.980447 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.980432 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 20:59:17.988986 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.988956 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wlcj\" (UniqueName: \"kubernetes.io/projected/9944dd1f-a9ea-4a6b-8174-2330011580a8-kube-api-access-2wlcj\") pod \"authorino-59f9b5458c-7mg4b\" (UID: \"9944dd1f-a9ea-4a6b-8174-2330011580a8\") " pod="kuadrant-system/authorino-59f9b5458c-7mg4b" Jul 29 20:59:17.989099 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.989003 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9944dd1f-a9ea-4a6b-8174-2330011580a8-tls-cert\") pod \"authorino-59f9b5458c-7mg4b\" (UID: \"9944dd1f-a9ea-4a6b-8174-2330011580a8\") " pod="kuadrant-system/authorino-59f9b5458c-7mg4b" Jul 29 20:59:17.989099 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:17.989091 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b494\" (UniqueName: \"kubernetes.io/projected/5aa9de95-4785-4ec9-a9ab-be931e6ee65f-kube-api-access-6b494\") pod \"authorino-58b6dd77c5-pm22m\" (UID: \"5aa9de95-4785-4ec9-a9ab-be931e6ee65f\") " pod="kuadrant-system/authorino-58b6dd77c5-pm22m" Jul 29 20:59:18.089468 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.089443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9944dd1f-a9ea-4a6b-8174-2330011580a8-tls-cert\") pod \"authorino-59f9b5458c-7mg4b\" (UID: \"9944dd1f-a9ea-4a6b-8174-2330011580a8\") " pod="kuadrant-system/authorino-59f9b5458c-7mg4b" Jul 29 20:59:18.089595 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.089496 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b494\" (UniqueName: \"kubernetes.io/projected/5aa9de95-4785-4ec9-a9ab-be931e6ee65f-kube-api-access-6b494\") pod \"authorino-58b6dd77c5-pm22m\" (UID: \"5aa9de95-4785-4ec9-a9ab-be931e6ee65f\") " pod="kuadrant-system/authorino-58b6dd77c5-pm22m" Jul 29 20:59:18.089595 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.089544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wlcj\" (UniqueName: \"kubernetes.io/projected/9944dd1f-a9ea-4a6b-8174-2330011580a8-kube-api-access-2wlcj\") pod \"authorino-59f9b5458c-7mg4b\" (UID: \"9944dd1f-a9ea-4a6b-8174-2330011580a8\") " pod="kuadrant-system/authorino-59f9b5458c-7mg4b" Jul 29 20:59:18.092011 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.091990 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9944dd1f-a9ea-4a6b-8174-2330011580a8-tls-cert\") pod \"authorino-59f9b5458c-7mg4b\" (UID: \"9944dd1f-a9ea-4a6b-8174-2330011580a8\") " pod="kuadrant-system/authorino-59f9b5458c-7mg4b" Jul 29 20:59:18.099733 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.099307 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b494\" (UniqueName: \"kubernetes.io/projected/5aa9de95-4785-4ec9-a9ab-be931e6ee65f-kube-api-access-6b494\") pod \"authorino-58b6dd77c5-pm22m\" (UID: \"5aa9de95-4785-4ec9-a9ab-be931e6ee65f\") " pod="kuadrant-system/authorino-58b6dd77c5-pm22m" Jul 29 20:59:18.099848 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.099462 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wlcj\" (UniqueName: \"kubernetes.io/projected/9944dd1f-a9ea-4a6b-8174-2330011580a8-kube-api-access-2wlcj\") pod \"authorino-59f9b5458c-7mg4b\" (UID: \"9944dd1f-a9ea-4a6b-8174-2330011580a8\") " pod="kuadrant-system/authorino-59f9b5458c-7mg4b" Jul 29 20:59:18.261801 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.261757 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59f9b5458c-7mg4b" Jul 29 20:59:18.264243 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.264210 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ctpwj" event={"ID":"f5709b9a-b593-4817-b254-78733fc5dbce","Type":"ContainerStarted","Data":"dd72a4645b6c4cb9123bda5b07f787ee4468d0f5118f82ff5ed81bf650701e82"} Jul 29 20:59:18.264370 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.264238 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b6dd77c5-pm22m" Jul 29 20:59:18.272228 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.272211 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b6dd77c5-pm22m" Jul 29 20:59:18.391643 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.391614 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b494\" (UniqueName: \"kubernetes.io/projected/5aa9de95-4785-4ec9-a9ab-be931e6ee65f-kube-api-access-6b494\") pod \"5aa9de95-4785-4ec9-a9ab-be931e6ee65f\" (UID: \"5aa9de95-4785-4ec9-a9ab-be931e6ee65f\") " Jul 29 20:59:18.393846 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.393820 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa9de95-4785-4ec9-a9ab-be931e6ee65f-kube-api-access-6b494" (OuterVolumeSpecName: "kube-api-access-6b494") pod "5aa9de95-4785-4ec9-a9ab-be931e6ee65f" (UID: "5aa9de95-4785-4ec9-a9ab-be931e6ee65f"). InnerVolumeSpecName "kube-api-access-6b494". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:59:18.412089 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.412067 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59f9b5458c-7mg4b"] Jul 29 20:59:18.414038 ip-10-0-137-160 kubenswrapper[2582]: W0729 20:59:18.414002 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9944dd1f_a9ea_4a6b_8174_2330011580a8.slice/crio-24c4b2dc26e350333836a08baf4f26342c32c93a7791db34a25b61b60576eae7 WatchSource:0}: Error finding container 24c4b2dc26e350333836a08baf4f26342c32c93a7791db34a25b61b60576eae7: Status 404 returned error can't find the container with id 24c4b2dc26e350333836a08baf4f26342c32c93a7791db34a25b61b60576eae7 Jul 29 20:59:18.492857 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:18.492832 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b494\" (UniqueName: \"kubernetes.io/projected/5aa9de95-4785-4ec9-a9ab-be931e6ee65f-kube-api-access-6b494\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:59:19.271256 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.271220 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59f9b5458c-7mg4b" event={"ID":"9944dd1f-a9ea-4a6b-8174-2330011580a8","Type":"ContainerStarted","Data":"f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055"} Jul 29 20:59:19.271730 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.271269 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59f9b5458c-7mg4b" event={"ID":"9944dd1f-a9ea-4a6b-8174-2330011580a8","Type":"ContainerStarted","Data":"24c4b2dc26e350333836a08baf4f26342c32c93a7791db34a25b61b60576eae7"} Jul 29 20:59:19.272852 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.272824 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ctpwj" event={"ID":"f5709b9a-b593-4817-b254-78733fc5dbce","Type":"ContainerStarted","Data":"845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11"} Jul 29 20:59:19.272956 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.272849 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-ctpwj" podUID="f5709b9a-b593-4817-b254-78733fc5dbce" containerName="authorino" containerID="cri-o://845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11" gracePeriod=30 Jul 29 20:59:19.272956 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.272899 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b6dd77c5-pm22m" Jul 29 20:59:19.292904 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.292862 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59f9b5458c-7mg4b" podStartSLOduration=1.9216598459999998 podStartE2EDuration="2.292834658s" totalImagesPullingTime="371.174812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:59:19.290601829 +0000 UTC m=+707.414191669" watchObservedRunningTime="2026-07-29 20:59:19.292834658 +0000 UTC m=+707.416424500" Jul 29 20:59:19.306181 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.306127 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-ctpwj" podStartSLOduration=1.834141085 podStartE2EDuration="2.306108201s" totalImagesPullingTime="471.967116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:59:19.304809628 +0000 UTC m=+707.428399477" watchObservedRunningTime="2026-07-29 20:59:19.306108201 +0000 UTC m=+707.429698044" Jul 29 20:59:19.322501 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.322449 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dggcf"] Jul 29 20:59:19.322742 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.322716 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-dggcf" podUID="61cc5ea7-861b-4782-a5af-d4c31d529a22" containerName="authorino" containerID="cri-o://e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6" gracePeriod=30 Jul 29 20:59:19.336679 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.336630 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58b6dd77c5-pm22m"] Jul 29 20:59:19.340077 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.340053 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58b6dd77c5-pm22m"] Jul 29 20:59:19.628515 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.628488 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ctpwj" Jul 29 20:59:19.645850 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.645820 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dggcf" Jul 29 20:59:19.703174 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.703139 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mfzv\" (UniqueName: \"kubernetes.io/projected/61cc5ea7-861b-4782-a5af-d4c31d529a22-kube-api-access-2mfzv\") pod \"61cc5ea7-861b-4782-a5af-d4c31d529a22\" (UID: \"61cc5ea7-861b-4782-a5af-d4c31d529a22\") " Jul 29 20:59:19.703326 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.703189 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76d7f\" (UniqueName: \"kubernetes.io/projected/f5709b9a-b593-4817-b254-78733fc5dbce-kube-api-access-76d7f\") pod \"f5709b9a-b593-4817-b254-78733fc5dbce\" (UID: \"f5709b9a-b593-4817-b254-78733fc5dbce\") " Jul 29 20:59:19.705379 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.705354 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5709b9a-b593-4817-b254-78733fc5dbce-kube-api-access-76d7f" (OuterVolumeSpecName: "kube-api-access-76d7f") pod "f5709b9a-b593-4817-b254-78733fc5dbce" (UID: "f5709b9a-b593-4817-b254-78733fc5dbce"). InnerVolumeSpecName "kube-api-access-76d7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:59:19.705588 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.705557 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cc5ea7-861b-4782-a5af-d4c31d529a22-kube-api-access-2mfzv" (OuterVolumeSpecName: "kube-api-access-2mfzv") pod "61cc5ea7-861b-4782-a5af-d4c31d529a22" (UID: "61cc5ea7-861b-4782-a5af-d4c31d529a22"). InnerVolumeSpecName "kube-api-access-2mfzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:59:19.804125 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.804013 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mfzv\" (UniqueName: \"kubernetes.io/projected/61cc5ea7-861b-4782-a5af-d4c31d529a22-kube-api-access-2mfzv\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:59:19.804241 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:19.804135 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76d7f\" (UniqueName: \"kubernetes.io/projected/f5709b9a-b593-4817-b254-78733fc5dbce-kube-api-access-76d7f\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 20:59:20.278758 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.278724 2582 generic.go:356] "Generic (PLEG): container finished" podID="61cc5ea7-861b-4782-a5af-d4c31d529a22" containerID="e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6" exitCode=0 Jul 29 20:59:20.279262 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.278772 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dggcf" Jul 29 20:59:20.279262 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.278808 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dggcf" event={"ID":"61cc5ea7-861b-4782-a5af-d4c31d529a22","Type":"ContainerDied","Data":"e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6"} Jul 29 20:59:20.279262 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.278844 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dggcf" event={"ID":"61cc5ea7-861b-4782-a5af-d4c31d529a22","Type":"ContainerDied","Data":"ec60c93c93d4be87524728b4e17c9458dc905a2835b8a10ee0c56ad56d4b96ff"} Jul 29 20:59:20.279262 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.278861 2582 scope.go:117] "RemoveContainer" containerID="e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6" Jul 29 20:59:20.280261 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.280242 2582 generic.go:356] "Generic (PLEG): container finished" podID="f5709b9a-b593-4817-b254-78733fc5dbce" containerID="845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11" exitCode=0 Jul 29 20:59:20.280348 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.280282 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ctpwj" Jul 29 20:59:20.280348 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.280322 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ctpwj" event={"ID":"f5709b9a-b593-4817-b254-78733fc5dbce","Type":"ContainerDied","Data":"845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11"} Jul 29 20:59:20.280446 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.280354 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ctpwj" event={"ID":"f5709b9a-b593-4817-b254-78733fc5dbce","Type":"ContainerDied","Data":"dd72a4645b6c4cb9123bda5b07f787ee4468d0f5118f82ff5ed81bf650701e82"} Jul 29 20:59:20.288803 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.288581 2582 scope.go:117] "RemoveContainer" containerID="e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6" Jul 29 20:59:20.289058 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:59:20.289004 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6\": container with ID starting with e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6 not found: ID does not exist" containerID="e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6" Jul 29 20:59:20.289140 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.289066 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6"} err="failed to get container status \"e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6\": rpc error: code = NotFound desc = could not find container \"e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6\": container with ID starting with e3cd64c4294983e61cee9379b3e36bcdd943480e8fd489bb00bbef8dc60159f6 not found: ID does not exist" Jul 29 20:59:20.289140 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.289083 2582 scope.go:117] "RemoveContainer" containerID="845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11" Jul 29 20:59:20.297119 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.297101 2582 scope.go:117] "RemoveContainer" containerID="845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11" Jul 29 20:59:20.297382 ip-10-0-137-160 kubenswrapper[2582]: E0729 20:59:20.297362 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11\": container with ID starting with 845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11 not found: ID does not exist" containerID="845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11" Jul 29 20:59:20.297419 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.297389 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11"} err="failed to get container status \"845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11\": rpc error: code = NotFound desc = could not find container \"845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11\": container with ID starting with 845070164c39e75e6c8da8e9371ae058ecb244bf4e86cf00aee465b67e435c11 not found: ID does not exist" Jul 29 20:59:20.304643 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.304625 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dggcf"] Jul 29 20:59:20.308696 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.308665 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dggcf"] Jul 29 20:59:20.320853 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.320834 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ctpwj"] Jul 29 20:59:20.322535 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.322519 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ctpwj"] Jul 29 20:59:20.493231 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.493205 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aa9de95-4785-4ec9-a9ab-be931e6ee65f" path="/var/lib/kubelet/pods/5aa9de95-4785-4ec9-a9ab-be931e6ee65f/volumes" Jul 29 20:59:20.493457 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.493439 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61cc5ea7-861b-4782-a5af-d4c31d529a22" path="/var/lib/kubelet/pods/61cc5ea7-861b-4782-a5af-d4c31d529a22/volumes" Jul 29 20:59:20.493756 ip-10-0-137-160 kubenswrapper[2582]: I0729 20:59:20.493743 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5709b9a-b593-4817-b254-78733fc5dbce" path="/var/lib/kubelet/pods/f5709b9a-b593-4817-b254-78733fc5dbce/volumes" Jul 29 21:00:02.756449 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:02.756416 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:00:10.529110 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:10.529082 2582 generic.go:356] "Generic (PLEG): container finished" podID="07571bcc-6ebe-4a1b-8749-f072d9e5f6a9" containerID="5b818452d128e28541d8e66f836ea3624764e9235d92a8ddab8bf7a44b64d4e1" exitCode=0 Jul 29 21:00:10.529515 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:10.529128 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" event={"ID":"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9","Type":"ContainerDied","Data":"5b818452d128e28541d8e66f836ea3624764e9235d92a8ddab8bf7a44b64d4e1"} Jul 29 21:00:10.529515 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:10.529446 2582 scope.go:117] "RemoveContainer" containerID="5b818452d128e28541d8e66f836ea3624764e9235d92a8ddab8bf7a44b64d4e1" Jul 29 21:00:10.535881 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:10.530819 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:00:10.535881 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:10.530996 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:00:10.535881 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:10.531144 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:00:11.535469 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:11.535434 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" event={"ID":"07571bcc-6ebe-4a1b-8749-f072d9e5f6a9","Type":"ContainerStarted","Data":"7a1d4d522eb91118efbff6a07bd129e023f99b117190714999240ee538bc0973"} Jul 29 21:00:11.614626 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:11.614483 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 21:00:11.615554 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:11.615522 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-qbhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 29 21:00:11.615815 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:11.615779 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" podUID="07571bcc-6ebe-4a1b-8749-f072d9e5f6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 29 21:00:12.614125 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:12.614084 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-qbhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 29 21:00:12.614628 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:12.614151 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" podUID="07571bcc-6ebe-4a1b-8749-f072d9e5f6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 29 21:00:13.614194 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:13.614154 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-qbhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 29 21:00:13.614777 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:13.614749 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" podUID="07571bcc-6ebe-4a1b-8749-f072d9e5f6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 29 21:00:14.069011 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:14.068966 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 21:00:14.614453 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:14.614409 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-qbhbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 29 21:00:14.614962 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:14.614481 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" podUID="07571bcc-6ebe-4a1b-8749-f072d9e5f6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 29 21:00:15.617452 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:15.617428 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 21:00:15.618493 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:15.618471 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-qbhbs" Jul 29 21:00:26.945120 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:26.945073 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:00:46.741719 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:46.741687 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:00:52.758190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:00:52.758151 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:01:15.455855 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:01:15.455820 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:01:23.751592 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:01:23.751555 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:01:35.351450 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:01:35.351416 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:02:28.380160 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.380082 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d5759c778-m8bnm"] Jul 29 21:02:28.380690 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.380667 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5709b9a-b593-4817-b254-78733fc5dbce" containerName="authorino" Jul 29 21:02:28.380772 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.380696 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5709b9a-b593-4817-b254-78733fc5dbce" containerName="authorino" Jul 29 21:02:28.380772 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.380721 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61cc5ea7-861b-4782-a5af-d4c31d529a22" containerName="authorino" Jul 29 21:02:28.380772 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.380729 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="61cc5ea7-861b-4782-a5af-d4c31d529a22" containerName="authorino" Jul 29 21:02:28.380931 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.380834 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="61cc5ea7-861b-4782-a5af-d4c31d529a22" containerName="authorino" Jul 29 21:02:28.380931 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.380847 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5709b9a-b593-4817-b254-78733fc5dbce" containerName="authorino" Jul 29 21:02:28.384132 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.384116 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5759c778-m8bnm" Jul 29 21:02:28.391634 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.391605 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d5759c778-m8bnm"] Jul 29 21:02:28.517870 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.517845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-689w2\" (UniqueName: \"kubernetes.io/projected/30e6ac2d-e619-4729-b687-771df9b4ab32-kube-api-access-689w2\") pod \"authorino-d5759c778-m8bnm\" (UID: \"30e6ac2d-e619-4729-b687-771df9b4ab32\") " pod="kuadrant-system/authorino-d5759c778-m8bnm" Jul 29 21:02:28.517996 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.517894 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/30e6ac2d-e619-4729-b687-771df9b4ab32-tls-cert\") pod \"authorino-d5759c778-m8bnm\" (UID: \"30e6ac2d-e619-4729-b687-771df9b4ab32\") " pod="kuadrant-system/authorino-d5759c778-m8bnm" Jul 29 21:02:28.619068 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.619016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-689w2\" (UniqueName: \"kubernetes.io/projected/30e6ac2d-e619-4729-b687-771df9b4ab32-kube-api-access-689w2\") pod \"authorino-d5759c778-m8bnm\" (UID: \"30e6ac2d-e619-4729-b687-771df9b4ab32\") " pod="kuadrant-system/authorino-d5759c778-m8bnm" Jul 29 21:02:28.619191 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.619082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/30e6ac2d-e619-4729-b687-771df9b4ab32-tls-cert\") pod \"authorino-d5759c778-m8bnm\" (UID: \"30e6ac2d-e619-4729-b687-771df9b4ab32\") " pod="kuadrant-system/authorino-d5759c778-m8bnm" Jul 29 21:02:28.621595 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.621568 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/30e6ac2d-e619-4729-b687-771df9b4ab32-tls-cert\") pod \"authorino-d5759c778-m8bnm\" (UID: \"30e6ac2d-e619-4729-b687-771df9b4ab32\") " pod="kuadrant-system/authorino-d5759c778-m8bnm" Jul 29 21:02:28.629870 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.629852 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-689w2\" (UniqueName: \"kubernetes.io/projected/30e6ac2d-e619-4729-b687-771df9b4ab32-kube-api-access-689w2\") pod \"authorino-d5759c778-m8bnm\" (UID: \"30e6ac2d-e619-4729-b687-771df9b4ab32\") " pod="kuadrant-system/authorino-d5759c778-m8bnm" Jul 29 21:02:28.713292 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.713213 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5759c778-m8bnm" Jul 29 21:02:28.846997 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:28.846976 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d5759c778-m8bnm"] Jul 29 21:02:28.848234 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:02:28.848206 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30e6ac2d_e619_4729_b687_771df9b4ab32.slice/crio-4f2fdec09adbb57c6f3f7af6a44fa879b638e8153053be653dbc23ded91634aa WatchSource:0}: Error finding container 4f2fdec09adbb57c6f3f7af6a44fa879b638e8153053be653dbc23ded91634aa: Status 404 returned error can't find the container with id 4f2fdec09adbb57c6f3f7af6a44fa879b638e8153053be653dbc23ded91634aa Jul 29 21:02:29.209366 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:29.209330 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d5759c778-m8bnm" event={"ID":"30e6ac2d-e619-4729-b687-771df9b4ab32","Type":"ContainerStarted","Data":"4f2fdec09adbb57c6f3f7af6a44fa879b638e8153053be653dbc23ded91634aa"} Jul 29 21:02:30.216876 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:30.216838 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d5759c778-m8bnm" event={"ID":"30e6ac2d-e619-4729-b687-771df9b4ab32","Type":"ContainerStarted","Data":"5da77809a463b70a05620a1a1fbaf502456b53153296dd4a078fc1c4ef9fd46b"} Jul 29 21:02:30.234562 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:30.234510 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d5759c778-m8bnm" podStartSLOduration=1.7577437740000001 podStartE2EDuration="2.234491498s" totalImagesPullingTime="476.747724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:02:30.232280231 +0000 UTC m=+898.355870075" watchObservedRunningTime="2026-07-29 21:02:30.234491498 +0000 UTC m=+898.358081340" Jul 29 21:02:30.262509 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:30.262477 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59f9b5458c-7mg4b"] Jul 29 21:02:30.262709 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:30.262684 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59f9b5458c-7mg4b" podUID="9944dd1f-a9ea-4a6b-8174-2330011580a8" containerName="authorino" containerID="cri-o://f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055" gracePeriod=30 Jul 29 21:02:30.528323 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:30.528295 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59f9b5458c-7mg4b" Jul 29 21:02:30.637538 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:30.637510 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9944dd1f-a9ea-4a6b-8174-2330011580a8-tls-cert\") pod \"9944dd1f-a9ea-4a6b-8174-2330011580a8\" (UID: \"9944dd1f-a9ea-4a6b-8174-2330011580a8\") " Jul 29 21:02:30.637716 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:30.637587 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wlcj\" (UniqueName: \"kubernetes.io/projected/9944dd1f-a9ea-4a6b-8174-2330011580a8-kube-api-access-2wlcj\") pod \"9944dd1f-a9ea-4a6b-8174-2330011580a8\" (UID: \"9944dd1f-a9ea-4a6b-8174-2330011580a8\") " Jul 29 21:02:30.639871 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:30.639840 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9944dd1f-a9ea-4a6b-8174-2330011580a8-kube-api-access-2wlcj" (OuterVolumeSpecName: "kube-api-access-2wlcj") pod "9944dd1f-a9ea-4a6b-8174-2330011580a8" (UID: "9944dd1f-a9ea-4a6b-8174-2330011580a8"). InnerVolumeSpecName "kube-api-access-2wlcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:02:30.647982 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:30.647955 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9944dd1f-a9ea-4a6b-8174-2330011580a8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9944dd1f-a9ea-4a6b-8174-2330011580a8" (UID: "9944dd1f-a9ea-4a6b-8174-2330011580a8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 21:02:30.739004 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:30.738962 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wlcj\" (UniqueName: \"kubernetes.io/projected/9944dd1f-a9ea-4a6b-8174-2330011580a8-kube-api-access-2wlcj\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:02:30.739004 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:30.739003 2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9944dd1f-a9ea-4a6b-8174-2330011580a8-tls-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:02:31.224056 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:31.224006 2582 generic.go:356] "Generic (PLEG): container finished" podID="9944dd1f-a9ea-4a6b-8174-2330011580a8" containerID="f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055" exitCode=0 Jul 29 21:02:31.224544 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:31.224080 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59f9b5458c-7mg4b" event={"ID":"9944dd1f-a9ea-4a6b-8174-2330011580a8","Type":"ContainerDied","Data":"f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055"} Jul 29 21:02:31.224544 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:31.224124 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59f9b5458c-7mg4b" event={"ID":"9944dd1f-a9ea-4a6b-8174-2330011580a8","Type":"ContainerDied","Data":"24c4b2dc26e350333836a08baf4f26342c32c93a7791db34a25b61b60576eae7"} Jul 29 21:02:31.224544 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:31.224150 2582 scope.go:117] "RemoveContainer" containerID="f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055" Jul 29 21:02:31.224544 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:31.224088 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59f9b5458c-7mg4b" Jul 29 21:02:31.233056 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:31.233012 2582 scope.go:117] "RemoveContainer" containerID="f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055" Jul 29 21:02:31.233315 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:02:31.233296 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055\": container with ID starting with f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055 not found: ID does not exist" containerID="f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055" Jul 29 21:02:31.233392 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:31.233325 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055"} err="failed to get container status \"f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055\": rpc error: code = NotFound desc = could not find container \"f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055\": container with ID starting with f5b45b7e24865d7a2617d625cc36ec6ad25a8a4ea363293bf4802e2bc7e48055 not found: ID does not exist" Jul 29 21:02:31.250058 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:31.250008 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59f9b5458c-7mg4b"] Jul 29 21:02:31.257688 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:31.257667 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59f9b5458c-7mg4b"] Jul 29 21:02:32.493505 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:32.493474 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9944dd1f-a9ea-4a6b-8174-2330011580a8" path="/var/lib/kubelet/pods/9944dd1f-a9ea-4a6b-8174-2330011580a8/volumes" Jul 29 21:02:59.362599 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:02:59.362561 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:03:09.754089 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:03:09.754052 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:03:19.958040 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:03:19.957983 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:05:02.849379 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:05:02.849334 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:05:11.857612 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:05:11.857573 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:05:40.485813 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:05:40.485720 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:05:49.951709 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:05:49.951676 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:05:58.852234 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:05:58.852195 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:06:10.252532 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:06:10.252494 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:06:21.462411 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:06:21.462374 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:06:30.952534 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:06:30.952499 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:06:42.250276 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:06:42.250238 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:06:52.250011 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:06:52.249971 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:07:17.257338 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:07:17.257257 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:07:27.151539 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:07:27.151502 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:07:50.958319 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:07:50.958279 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:08:02.055062 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:08:02.055004 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:08:11.857268 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:08:11.857229 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:08:28.562556 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:08:28.562465 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:08:39.160323 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:08:39.160290 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:08:48.957963 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:08:48.957931 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:08:59.055531 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:08:59.055493 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:09:09.046528 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:09.046491 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:09:33.727382 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:33.727343 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx"] Jul 29 21:09:33.727973 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:33.727661 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" podUID="a1ceae93-dc20-4cd6-b15d-aba70fc993dc" containerName="manager" containerID="cri-o://b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c" gracePeriod=10 Jul 29 21:09:38.081517 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.081489 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" Jul 29 21:09:38.197586 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.197480 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq6hn\" (UniqueName: \"kubernetes.io/projected/a1ceae93-dc20-4cd6-b15d-aba70fc993dc-kube-api-access-fq6hn\") pod \"a1ceae93-dc20-4cd6-b15d-aba70fc993dc\" (UID: \"a1ceae93-dc20-4cd6-b15d-aba70fc993dc\") " Jul 29 21:09:38.197586 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.197525 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1ceae93-dc20-4cd6-b15d-aba70fc993dc-extensions-socket-volume\") pod \"a1ceae93-dc20-4cd6-b15d-aba70fc993dc\" (UID: \"a1ceae93-dc20-4cd6-b15d-aba70fc993dc\") " Jul 29 21:09:38.197930 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.197904 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ceae93-dc20-4cd6-b15d-aba70fc993dc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a1ceae93-dc20-4cd6-b15d-aba70fc993dc" (UID: "a1ceae93-dc20-4cd6-b15d-aba70fc993dc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:09:38.199814 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.199783 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ceae93-dc20-4cd6-b15d-aba70fc993dc-kube-api-access-fq6hn" (OuterVolumeSpecName: "kube-api-access-fq6hn") pod "a1ceae93-dc20-4cd6-b15d-aba70fc993dc" (UID: "a1ceae93-dc20-4cd6-b15d-aba70fc993dc"). InnerVolumeSpecName "kube-api-access-fq6hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:09:38.298456 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.298411 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq6hn\" (UniqueName: \"kubernetes.io/projected/a1ceae93-dc20-4cd6-b15d-aba70fc993dc-kube-api-access-fq6hn\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:09:38.298456 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.298448 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1ceae93-dc20-4cd6-b15d-aba70fc993dc-extensions-socket-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:09:38.385943 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.385900 2582 generic.go:356] "Generic (PLEG): container finished" podID="a1ceae93-dc20-4cd6-b15d-aba70fc993dc" containerID="b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c" exitCode=0 Jul 29 21:09:38.386164 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.385956 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" Jul 29 21:09:38.386164 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.385979 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" event={"ID":"a1ceae93-dc20-4cd6-b15d-aba70fc993dc","Type":"ContainerDied","Data":"b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c"} Jul 29 21:09:38.386164 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.386069 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx" event={"ID":"a1ceae93-dc20-4cd6-b15d-aba70fc993dc","Type":"ContainerDied","Data":"2861e22a6c40e11ccae15fd6866ecb52d642ec8a3dd3b54abcba25e11912ec80"} Jul 29 21:09:38.386164 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.386092 2582 scope.go:117] "RemoveContainer" containerID="b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c" Jul 29 21:09:38.396458 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.396441 2582 scope.go:117] "RemoveContainer" containerID="b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c" Jul 29 21:09:38.396725 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:09:38.396705 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c\": container with ID starting with b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c not found: ID does not exist" containerID="b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c" Jul 29 21:09:38.396792 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.396734 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c"} err="failed to get container status \"b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c\": rpc error: code = NotFound desc = could not find container \"b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c\": container with ID starting with b61c00ccfc18b50cd80facbf56a623fa5b82bfe35728ef4f6875a8c7c248f03c not found: ID does not exist" Jul 29 21:09:38.408611 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.408580 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx"] Jul 29 21:09:38.411150 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.411128 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m57rx"] Jul 29 21:09:38.492629 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:09:38.492600 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1ceae93-dc20-4cd6-b15d-aba70fc993dc" path="/var/lib/kubelet/pods/a1ceae93-dc20-4cd6-b15d-aba70fc993dc/volumes" Jul 29 21:10:35.014287 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.014246 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz"] Jul 29 21:10:35.014740 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.014590 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9944dd1f-a9ea-4a6b-8174-2330011580a8" containerName="authorino" Jul 29 21:10:35.014740 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.014601 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9944dd1f-a9ea-4a6b-8174-2330011580a8" containerName="authorino" Jul 29 21:10:35.014740 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.014622 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ceae93-dc20-4cd6-b15d-aba70fc993dc" containerName="manager" Jul 29 21:10:35.014740 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.014627 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ceae93-dc20-4cd6-b15d-aba70fc993dc" containerName="manager" Jul 29 21:10:35.014740 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.014680 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9944dd1f-a9ea-4a6b-8174-2330011580a8" containerName="authorino" Jul 29 21:10:35.014740 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.014690 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1ceae93-dc20-4cd6-b15d-aba70fc993dc" containerName="manager" Jul 29 21:10:35.018906 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.018886 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" Jul 29 21:10:35.022100 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.022073 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wdf79\"" Jul 29 21:10:35.044963 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.044933 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz"] Jul 29 21:10:35.086038 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.085982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rvc\" (UniqueName: \"kubernetes.io/projected/d6d56a11-13e1-4a78-876b-a56f7267813c-kube-api-access-64rvc\") pod \"kuadrant-operator-controller-manager-86878c769f-wj4cz\" (UID: \"d6d56a11-13e1-4a78-876b-a56f7267813c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" Jul 29 21:10:35.086234 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.086092 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6d56a11-13e1-4a78-876b-a56f7267813c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wj4cz\" (UID: \"d6d56a11-13e1-4a78-876b-a56f7267813c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" Jul 29 21:10:35.187332 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.187291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rvc\" (UniqueName: \"kubernetes.io/projected/d6d56a11-13e1-4a78-876b-a56f7267813c-kube-api-access-64rvc\") pod \"kuadrant-operator-controller-manager-86878c769f-wj4cz\" (UID: \"d6d56a11-13e1-4a78-876b-a56f7267813c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" Jul 29 21:10:35.187521 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.187339 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6d56a11-13e1-4a78-876b-a56f7267813c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wj4cz\" (UID: \"d6d56a11-13e1-4a78-876b-a56f7267813c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" Jul 29 21:10:35.187675 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.187659 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6d56a11-13e1-4a78-876b-a56f7267813c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wj4cz\" (UID: \"d6d56a11-13e1-4a78-876b-a56f7267813c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" Jul 29 21:10:35.201397 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.201364 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rvc\" (UniqueName: \"kubernetes.io/projected/d6d56a11-13e1-4a78-876b-a56f7267813c-kube-api-access-64rvc\") pod \"kuadrant-operator-controller-manager-86878c769f-wj4cz\" (UID: \"d6d56a11-13e1-4a78-876b-a56f7267813c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" Jul 29 21:10:35.329243 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.329153 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" Jul 29 21:10:35.506853 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.506826 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz"] Jul 29 21:10:35.509532 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:10:35.509505 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d56a11_13e1_4a78_876b_a56f7267813c.slice/crio-97ae76fc632f734bc838c68a22ebcd6f78e5555df76dec759edfd31fa0ff1ba8 WatchSource:0}: Error finding container 97ae76fc632f734bc838c68a22ebcd6f78e5555df76dec759edfd31fa0ff1ba8: Status 404 returned error can't find the container with id 97ae76fc632f734bc838c68a22ebcd6f78e5555df76dec759edfd31fa0ff1ba8 Jul 29 21:10:35.670415 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.670311 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" event={"ID":"d6d56a11-13e1-4a78-876b-a56f7267813c","Type":"ContainerStarted","Data":"412241d6d599d5badedf9d80242c471fe55c6a8cc4f4b2739d689e2dd0597afc"} Jul 29 21:10:35.670415 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.670372 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" Jul 29 21:10:35.670415 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.670389 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" event={"ID":"d6d56a11-13e1-4a78-876b-a56f7267813c","Type":"ContainerStarted","Data":"97ae76fc632f734bc838c68a22ebcd6f78e5555df76dec759edfd31fa0ff1ba8"} Jul 29 21:10:35.710573 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:35.710517 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" podStartSLOduration=1.710499448 podStartE2EDuration="1.710499448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:10:35.709449141 +0000 UTC m=+1383.833038982" watchObservedRunningTime="2026-07-29 21:10:35.710499448 +0000 UTC m=+1383.834089291" Jul 29 21:10:46.677402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:46.677371 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4cz" Jul 29 21:10:56.435975 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:10:56.435931 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:11:08.699655 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:11:08.699610 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:11:21.297547 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:11:21.297507 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:11:36.859242 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:11:36.859141 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:11:48.455920 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:11:48.455885 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:11:56.966757 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:11:56.966717 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:12:06.656018 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:12:06.655980 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:12:17.656621 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:12:17.656587 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:12:26.663154 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:12:26.663121 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:12:38.362171 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:12:38.362133 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:12:49.549913 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:12:49.549876 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:12:57.261346 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:12:57.261251 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:13:14.252179 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:13:14.252139 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:13:24.557681 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:13:24.557643 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:13:36.855609 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:13:36.855569 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:13:49.149165 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:13:49.149124 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:14:17.456571 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:14:17.456539 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:14:31.956004 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:14:31.955908 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:14:38.347666 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:14:38.347619 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:14:54.561179 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:14:54.561142 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:15:00.135742 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:00.135703 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755995-k45pd"] Jul 29 21:15:00.139013 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:00.138996 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-k45pd" Jul 29 21:15:00.141625 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:00.141606 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\"" Jul 29 21:15:00.146793 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:00.146761 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755995-k45pd"] Jul 29 21:15:00.150112 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:00.150092 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-k45pd" Jul 29 21:15:00.289835 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:00.289812 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755995-k45pd"] Jul 29 21:15:00.291474 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:15:00.291441 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f06ae2_cdd2_405c_8060_49fd43a31a77.slice/crio-a25755c3798952877e001eb2a18fcdf60fe11b7512715e2adc1a282ae5a045d6 WatchSource:0}: Error finding container a25755c3798952877e001eb2a18fcdf60fe11b7512715e2adc1a282ae5a045d6: Status 404 returned error can't find the container with id a25755c3798952877e001eb2a18fcdf60fe11b7512715e2adc1a282ae5a045d6 Jul 29 21:15:00.293555 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:00.293534 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 21:15:01.021107 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:01.021073 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-k45pd" event={"ID":"91f06ae2-cdd2-405c-8060-49fd43a31a77","Type":"ContainerStarted","Data":"a25755c3798952877e001eb2a18fcdf60fe11b7512715e2adc1a282ae5a045d6"} Jul 29 21:15:06.245908 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:06.245871 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:15:14.091050 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:14.090993 2582 generic.go:356] "Generic (PLEG): container finished" podID="91f06ae2-cdd2-405c-8060-49fd43a31a77" containerID="a86f2f69bd515f5518173d231d7b95560eac54cf383a328ba5f252d541d49df3" exitCode=0 Jul 29 21:15:14.091464 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:14.091088 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-k45pd" event={"ID":"91f06ae2-cdd2-405c-8060-49fd43a31a77","Type":"ContainerDied","Data":"a86f2f69bd515f5518173d231d7b95560eac54cf383a328ba5f252d541d49df3"} Jul 29 21:15:15.238598 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:15.238575 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-k45pd" Jul 29 21:15:16.104143 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:16.104115 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-k45pd" Jul 29 21:15:16.104143 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:16.104138 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755995-k45pd" event={"ID":"91f06ae2-cdd2-405c-8060-49fd43a31a77","Type":"ContainerDied","Data":"a25755c3798952877e001eb2a18fcdf60fe11b7512715e2adc1a282ae5a045d6"} Jul 29 21:15:16.104370 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:16.104184 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a25755c3798952877e001eb2a18fcdf60fe11b7512715e2adc1a282ae5a045d6" Jul 29 21:15:16.548276 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:16.548238 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:15:24.950243 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:24.950207 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:15:35.051715 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:35.051675 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:15:44.556304 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:44.556260 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:15:57.253332 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:15:57.253254 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:16:07.252667 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:16:07.252629 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:16:17.359416 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:16:17.359375 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:16:27.349869 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:16:27.349830 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:16:40.749772 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:16:40.749733 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:16:55.252349 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:16:55.252311 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:17:06.254244 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:17:06.254200 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:17:24.266790 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:17:24.266691 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:17:34.180858 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:17:34.180818 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:17:48.658083 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:17:48.658044 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:17:58.388838 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:17:58.388802 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:18:08.857060 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:08.857001 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:18:09.222148 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.222064 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg"] Jul 29 21:18:09.222414 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.222402 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f06ae2-cdd2-405c-8060-49fd43a31a77" containerName="cleanup" Jul 29 21:18:09.222460 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.222417 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f06ae2-cdd2-405c-8060-49fd43a31a77" containerName="cleanup" Jul 29 21:18:09.222508 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.222499 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f06ae2-cdd2-405c-8060-49fd43a31a77" containerName="cleanup" Jul 29 21:18:09.225510 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.225490 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.256800 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.256769 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg"] Jul 29 21:18:09.397325 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.397279 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-istio-data\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.397565 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.397350 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-workload-certs\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.397565 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.397395 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-credential-socket\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.397565 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.397425 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-workload-socket\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.397565 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.397460 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0168312c-79e1-4a70-a3d2-0313a262d941-istio-podinfo\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.397565 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.397489 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0168312c-79e1-4a70-a3d2-0313a262d941-istio-token\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.397565 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.397515 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbb5l\" (UniqueName: \"kubernetes.io/projected/0168312c-79e1-4a70-a3d2-0313a262d941-kube-api-access-cbb5l\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.397565 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.397544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-istio-envoy\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.397963 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.397571 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0168312c-79e1-4a70-a3d2-0313a262d941-istiod-ca-cert\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.498489 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.498454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-istio-data\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.498680 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.498504 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-workload-certs\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.498680 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.498651 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-credential-socket\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.498800 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.498706 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-workload-socket\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.498800 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.498749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0168312c-79e1-4a70-a3d2-0313a262d941-istio-podinfo\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.498911 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.498807 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0168312c-79e1-4a70-a3d2-0313a262d941-istio-token\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.498911 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.498833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbb5l\" (UniqueName: \"kubernetes.io/projected/0168312c-79e1-4a70-a3d2-0313a262d941-kube-api-access-cbb5l\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.498911 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.498898 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-workload-certs\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.499056 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.498984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-credential-socket\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.499185 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.499163 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-workload-socket\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.499319 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.499294 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-istio-envoy\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.499479 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.499331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0168312c-79e1-4a70-a3d2-0313a262d941-istiod-ca-cert\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.499479 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.499472 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-istio-data\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.500141 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.500119 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0168312c-79e1-4a70-a3d2-0313a262d941-istiod-ca-cert\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.501731 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.501709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-istio-envoy\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.501999 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.501978 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0168312c-79e1-4a70-a3d2-0313a262d941-istio-podinfo\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.507471 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.507444 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0168312c-79e1-4a70-a3d2-0313a262d941-istio-token\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.507760 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.507741 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbb5l\" (UniqueName: \"kubernetes.io/projected/0168312c-79e1-4a70-a3d2-0313a262d941-kube-api-access-cbb5l\") pod \"e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.535801 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.535764 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:09.693104 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.693048 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg"] Jul 29 21:18:09.694854 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:18:09.694820 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0168312c_79e1_4a70_a3d2_0313a262d941.slice/crio-8ac4e7355712c5766c113af9ff5cc501f6584c61b2e9d6fa9b0e109daa9ca453 WatchSource:0}: Error finding container 8ac4e7355712c5766c113af9ff5cc501f6584c61b2e9d6fa9b0e109daa9ca453: Status 404 returned error can't find the container with id 8ac4e7355712c5766c113af9ff5cc501f6584c61b2e9d6fa9b0e109daa9ca453 Jul 29 21:18:09.697109 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.697073 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:18:09.697212 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.697148 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:18:09.697212 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:09.697183 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:18:10.004626 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:10.004512 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" event={"ID":"0168312c-79e1-4a70-a3d2-0313a262d941","Type":"ContainerStarted","Data":"e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46"} Jul 29 21:18:10.004626 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:10.004581 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" event={"ID":"0168312c-79e1-4a70-a3d2-0313a262d941","Type":"ContainerStarted","Data":"8ac4e7355712c5766c113af9ff5cc501f6584c61b2e9d6fa9b0e109daa9ca453"} Jul 29 21:18:10.025800 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:10.025746 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" podStartSLOduration=1.025729206 podStartE2EDuration="1.025729206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:10.023389265 +0000 UTC m=+1838.146979147" watchObservedRunningTime="2026-07-29 21:18:10.025729206 +0000 UTC m=+1838.149319047" Jul 29 21:18:10.536368 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:10.536328 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:10.541353 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:10.541323 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:11.010324 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:11.010296 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:11.011402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:11.011383 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:13.769841 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:13.769808 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64"] Jul 29 21:18:13.773545 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:13.773518 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" Jul 29 21:18:13.784235 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:13.784189 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dbfaf7e5\"" Jul 29 21:18:13.784402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:13.784316 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dbfaf7e5-dockercfg-79bdh\"" Jul 29 21:18:13.794652 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:13.794613 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64"] Jul 29 21:18:13.939898 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:13.939858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqp59\" (UniqueName: \"kubernetes.io/projected/19624591-d850-487c-bb6a-ae69682ac6fb-kube-api-access-sqp59\") pod \"payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64\" (UID: \"19624591-d850-487c-bb6a-ae69682ac6fb\") " pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" Jul 29 21:18:13.940120 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:13.939910 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19624591-d850-487c-bb6a-ae69682ac6fb-plugins-config-volume\") pod \"payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64\" (UID: \"19624591-d850-487c-bb6a-ae69682ac6fb\") " pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" Jul 29 21:18:14.040908 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:14.040812 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqp59\" (UniqueName: \"kubernetes.io/projected/19624591-d850-487c-bb6a-ae69682ac6fb-kube-api-access-sqp59\") pod \"payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64\" (UID: \"19624591-d850-487c-bb6a-ae69682ac6fb\") " pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" Jul 29 21:18:14.040908 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:14.040862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19624591-d850-487c-bb6a-ae69682ac6fb-plugins-config-volume\") pod \"payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64\" (UID: \"19624591-d850-487c-bb6a-ae69682ac6fb\") " pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" Jul 29 21:18:14.041669 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:14.041649 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19624591-d850-487c-bb6a-ae69682ac6fb-plugins-config-volume\") pod \"payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64\" (UID: \"19624591-d850-487c-bb6a-ae69682ac6fb\") " pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" Jul 29 21:18:14.052362 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:14.052338 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqp59\" (UniqueName: \"kubernetes.io/projected/19624591-d850-487c-bb6a-ae69682ac6fb-kube-api-access-sqp59\") pod \"payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64\" (UID: \"19624591-d850-487c-bb6a-ae69682ac6fb\") " pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" Jul 29 21:18:14.085360 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:14.085322 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" Jul 29 21:18:14.239402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:14.239376 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64"] Jul 29 21:18:14.241547 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:18:14.241518 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19624591_d850_487c_bb6a_ae69682ac6fb.slice/crio-90caed16d1760d7cdd0754c4f3b1a6e4a7aea90a037d3b3348f3ee62eb1888c5 WatchSource:0}: Error finding container 90caed16d1760d7cdd0754c4f3b1a6e4a7aea90a037d3b3348f3ee62eb1888c5: Status 404 returned error can't find the container with id 90caed16d1760d7cdd0754c4f3b1a6e4a7aea90a037d3b3348f3ee62eb1888c5 Jul 29 21:18:15.033708 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:15.033619 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" event={"ID":"19624591-d850-487c-bb6a-ae69682ac6fb","Type":"ContainerStarted","Data":"90caed16d1760d7cdd0754c4f3b1a6e4a7aea90a037d3b3348f3ee62eb1888c5"} Jul 29 21:18:16.040766 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:16.040725 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" event={"ID":"19624591-d850-487c-bb6a-ae69682ac6fb","Type":"ContainerStarted","Data":"5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c"} Jul 29 21:18:16.041166 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:16.040794 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" Jul 29 21:18:16.065478 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:16.065422 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" podStartSLOduration=1.5722235599999999 podStartE2EDuration="3.065406387s" totalImagesPullingTime="1.493182827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:16.063890147 +0000 UTC m=+1844.187479989" watchObservedRunningTime="2026-07-29 21:18:16.065406387 +0000 UTC m=+1844.188996228" Jul 29 21:18:19.572410 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:19.572370 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:18:27.047712 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:27.047677 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" Jul 29 21:18:27.053200 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:27.053168 2582 patch_prober.go:28] interesting pod/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 21:18:27.053299 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:27.053243 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 21:18:28.106915 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:28.106884 2582 generic.go:356] "Generic (PLEG): container finished" podID="0168312c-79e1-4a70-a3d2-0313a262d941" containerID="e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46" exitCode=0 Jul 29 21:18:28.107418 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:28.106937 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" event={"ID":"0168312c-79e1-4a70-a3d2-0313a262d941","Type":"ContainerDied","Data":"e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46"} Jul 29 21:18:28.107418 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:28.107319 2582 scope.go:117] "RemoveContainer" containerID="e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46" Jul 29 21:18:28.108406 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:28.108369 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:18:28.108495 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:28.108478 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:18:28.108533 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:28.108521 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:18:29.125249 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:29.125214 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" event={"ID":"0168312c-79e1-4a70-a3d2-0313a262d941","Type":"ContainerStarted","Data":"95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091"} Jul 29 21:18:29.536677 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:29.536634 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:29.536902 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:29.536878 2582 patch_prober.go:28] interesting pod/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 29 21:18:29.536946 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:29.536928 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 29 21:18:30.536723 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:30.536685 2582 patch_prober.go:28] interesting pod/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 29 21:18:30.537180 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:30.536744 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 29 21:18:31.537218 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:31.537180 2582 patch_prober.go:28] interesting pod/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 29 21:18:31.537645 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:31.537240 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 29 21:18:32.536884 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:32.536850 2582 patch_prober.go:28] interesting pod/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 29 21:18:32.537091 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:32.536914 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 29 21:18:33.539848 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:33.539818 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:33.540302 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:33.540062 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:33.540783 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:33.540767 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:37.010630 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:37.010590 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch"] Jul 29 21:18:37.015435 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:37.015412 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch" Jul 29 21:18:37.018146 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:37.018119 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\"" Jul 29 21:18:37.018257 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:37.018215 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:18:37.022849 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:37.022826 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch"] Jul 29 21:18:37.068452 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:37.068412 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea1b825-1526-4042-819a-08473349b097-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch\" (UID: \"cea1b825-1526-4042-819a-08473349b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch" Jul 29 21:18:37.169007 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:37.168975 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea1b825-1526-4042-819a-08473349b097-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch\" (UID: \"cea1b825-1526-4042-819a-08473349b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch" Jul 29 21:18:37.169620 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:37.169601 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea1b825-1526-4042-819a-08473349b097-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch\" (UID: \"cea1b825-1526-4042-819a-08473349b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch" Jul 29 21:18:37.329232 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:37.329132 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch" Jul 29 21:18:37.473524 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:37.473499 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch"] Jul 29 21:18:37.476131 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:18:37.476104 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea1b825_1526_4042_819a_08473349b097.slice/crio-c4eac77a59d718a0bb17660c730e7248a932d1aef2bd5b891d540fa084ca0dd2 WatchSource:0}: Error finding container c4eac77a59d718a0bb17660c730e7248a932d1aef2bd5b891d540fa084ca0dd2: Status 404 returned error can't find the container with id c4eac77a59d718a0bb17660c730e7248a932d1aef2bd5b891d540fa084ca0dd2 Jul 29 21:18:38.177058 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:38.177006 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch" event={"ID":"cea1b825-1526-4042-819a-08473349b097","Type":"ContainerStarted","Data":"13c09d33e6bd12a4ac807aa44af2a6dfb2d82169e300ec3c8c6969cc3a36779b"} Jul 29 21:18:38.177461 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:38.177077 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch" event={"ID":"cea1b825-1526-4042-819a-08473349b097","Type":"ContainerStarted","Data":"c4eac77a59d718a0bb17660c730e7248a932d1aef2bd5b891d540fa084ca0dd2"} Jul 29 21:18:38.195856 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:38.195775 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch" podStartSLOduration=2.195751838 podStartE2EDuration="2.195751838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:38.194748628 +0000 UTC m=+1866.318338471" watchObservedRunningTime="2026-07-29 21:18:38.195751838 +0000 UTC m=+1866.319341682" Jul 29 21:18:39.183953 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:39.183918 2582 generic.go:356] "Generic (PLEG): container finished" podID="cea1b825-1526-4042-819a-08473349b097" containerID="13c09d33e6bd12a4ac807aa44af2a6dfb2d82169e300ec3c8c6969cc3a36779b" exitCode=0 Jul 29 21:18:39.184359 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:39.183998 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch" event={"ID":"cea1b825-1526-4042-819a-08473349b097","Type":"ContainerDied","Data":"13c09d33e6bd12a4ac807aa44af2a6dfb2d82169e300ec3c8c6969cc3a36779b"} Jul 29 21:18:40.331201 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:40.331176 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch" Jul 29 21:18:40.400552 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:40.400520 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea1b825-1526-4042-819a-08473349b097-maas-api-service-ca\") pod \"cea1b825-1526-4042-819a-08473349b097\" (UID: \"cea1b825-1526-4042-819a-08473349b097\") " Jul 29 21:18:40.400917 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:40.400887 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea1b825-1526-4042-819a-08473349b097-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cea1b825-1526-4042-819a-08473349b097" (UID: "cea1b825-1526-4042-819a-08473349b097"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:18:40.501482 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:40.501453 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea1b825-1526-4042-819a-08473349b097-maas-api-service-ca\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:18:41.196554 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:41.196519 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch" event={"ID":"cea1b825-1526-4042-819a-08473349b097","Type":"ContainerDied","Data":"c4eac77a59d718a0bb17660c730e7248a932d1aef2bd5b891d540fa084ca0dd2"} Jul 29 21:18:41.196772 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:41.196578 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4eac77a59d718a0bb17660c730e7248a932d1aef2bd5b891d540fa084ca0dd2" Jul 29 21:18:41.196772 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:41.196534 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch" Jul 29 21:18:43.170629 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.170591 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64"] Jul 29 21:18:43.171153 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.170878 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" podUID="19624591-d850-487c-bb6a-ae69682ac6fb" containerName="payload-processing" containerID="cri-o://5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c" gracePeriod=30 Jul 29 21:18:43.327338 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.327302 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch"] Jul 29 21:18:43.330921 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.330889 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dbfaf7e5-kbfch"] Jul 29 21:18:43.441467 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.441442 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" Jul 29 21:18:43.482888 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.482855 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg"] Jul 29 21:18:43.483142 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.483093 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" containerID="cri-o://95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091" gracePeriod=30 Jul 29 21:18:43.526281 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.526250 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19624591-d850-487c-bb6a-ae69682ac6fb-plugins-config-volume\") pod \"19624591-d850-487c-bb6a-ae69682ac6fb\" (UID: \"19624591-d850-487c-bb6a-ae69682ac6fb\") " Jul 29 21:18:43.526463 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.526310 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqp59\" (UniqueName: \"kubernetes.io/projected/19624591-d850-487c-bb6a-ae69682ac6fb-kube-api-access-sqp59\") pod \"19624591-d850-487c-bb6a-ae69682ac6fb\" (UID: \"19624591-d850-487c-bb6a-ae69682ac6fb\") " Jul 29 21:18:43.526639 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.526616 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19624591-d850-487c-bb6a-ae69682ac6fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19624591-d850-487c-bb6a-ae69682ac6fb" (UID: "19624591-d850-487c-bb6a-ae69682ac6fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:18:43.528556 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.528533 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19624591-d850-487c-bb6a-ae69682ac6fb-kube-api-access-sqp59" (OuterVolumeSpecName: "kube-api-access-sqp59") pod "19624591-d850-487c-bb6a-ae69682ac6fb" (UID: "19624591-d850-487c-bb6a-ae69682ac6fb"). InnerVolumeSpecName "kube-api-access-sqp59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:18:43.627848 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.627811 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19624591-d850-487c-bb6a-ae69682ac6fb-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:18:43.627848 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:43.627841 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqp59\" (UniqueName: \"kubernetes.io/projected/19624591-d850-487c-bb6a-ae69682ac6fb-kube-api-access-sqp59\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:18:44.213932 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.213899 2582 generic.go:356] "Generic (PLEG): container finished" podID="19624591-d850-487c-bb6a-ae69682ac6fb" containerID="5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c" exitCode=0 Jul 29 21:18:44.214439 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.213945 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" Jul 29 21:18:44.214439 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.213991 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" event={"ID":"19624591-d850-487c-bb6a-ae69682ac6fb","Type":"ContainerDied","Data":"5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c"} Jul 29 21:18:44.214439 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.214069 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64" event={"ID":"19624591-d850-487c-bb6a-ae69682ac6fb","Type":"ContainerDied","Data":"90caed16d1760d7cdd0754c4f3b1a6e4a7aea90a037d3b3348f3ee62eb1888c5"} Jul 29 21:18:44.214439 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.214109 2582 scope.go:117] "RemoveContainer" containerID="5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c" Jul 29 21:18:44.223411 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.223395 2582 scope.go:117] "RemoveContainer" containerID="5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c" Jul 29 21:18:44.223664 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:18:44.223644 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c\": container with ID starting with 5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c not found: ID does not exist" containerID="5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c" Jul 29 21:18:44.223709 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.223673 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c"} err="failed to get container status \"5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c\": rpc error: code = NotFound desc = could not find container \"5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c\": container with ID starting with 5211c8ee8c306422ea2c8ea2d2b37df46c318abf97667d62e08a8730160b247c not found: ID does not exist" Jul 29 21:18:44.240592 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.238545 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64"] Jul 29 21:18:44.243554 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.243528 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dbfaf7e5-69b5d8d7d4-lgj64"] Jul 29 21:18:44.493088 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.493054 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19624591-d850-487c-bb6a-ae69682ac6fb" path="/var/lib/kubelet/pods/19624591-d850-487c-bb6a-ae69682ac6fb/volumes" Jul 29 21:18:44.493492 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.493472 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cea1b825-1526-4042-819a-08473349b097" path="/var/lib/kubelet/pods/cea1b825-1526-4042-819a-08473349b097/volumes" Jul 29 21:18:44.572903 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.572869 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx"] Jul 29 21:18:44.573259 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.573246 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19624591-d850-487c-bb6a-ae69682ac6fb" containerName="payload-processing" Jul 29 21:18:44.573316 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.573261 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="19624591-d850-487c-bb6a-ae69682ac6fb" containerName="payload-processing" Jul 29 21:18:44.573316 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.573271 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cea1b825-1526-4042-819a-08473349b097" containerName="revoke-keys" Jul 29 21:18:44.573316 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.573276 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cea1b825-1526-4042-819a-08473349b097" containerName="revoke-keys" Jul 29 21:18:44.573413 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.573352 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cea1b825-1526-4042-819a-08473349b097" containerName="revoke-keys" Jul 29 21:18:44.573413 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.573362 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="19624591-d850-487c-bb6a-ae69682ac6fb" containerName="payload-processing" Jul 29 21:18:44.577672 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.577655 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.580394 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.580375 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-9900cabf-gw-openshift-default-dockercfg-bx6t7\"" Jul 29 21:18:44.594352 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.594321 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx"] Jul 29 21:18:44.739255 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.739212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-workload-certs\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.739255 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.739262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-istio-data\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.739553 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.739285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rmd\" (UniqueName: \"kubernetes.io/projected/753af74e-2e36-4324-aeca-e627b40eaee5-kube-api-access-v8rmd\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.739553 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.739307 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753af74e-2e36-4324-aeca-e627b40eaee5-istio-podinfo\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.739553 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.739329 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753af74e-2e36-4324-aeca-e627b40eaee5-istio-token\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.739553 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.739391 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-istio-envoy\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.739553 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.739414 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-workload-socket\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.739720 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.739544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753af74e-2e36-4324-aeca-e627b40eaee5-istiod-ca-cert\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.739720 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.739580 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-credential-socket\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.840595 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.840501 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-istio-envoy\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.840595 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.840544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-workload-socket\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.840824 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.840610 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753af74e-2e36-4324-aeca-e627b40eaee5-istiod-ca-cert\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.840824 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.840640 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-credential-socket\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.840824 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.840678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-workload-certs\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.840824 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.840711 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-istio-data\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.840824 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.840733 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rmd\" (UniqueName: \"kubernetes.io/projected/753af74e-2e36-4324-aeca-e627b40eaee5-kube-api-access-v8rmd\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.840824 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.840758 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753af74e-2e36-4324-aeca-e627b40eaee5-istio-podinfo\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.840824 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.840780 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753af74e-2e36-4324-aeca-e627b40eaee5-istio-token\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.841207 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.841016 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-workload-socket\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.841356 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.841327 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-credential-socket\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.841488 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.841367 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-istio-data\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.841488 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.841432 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-workload-certs\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.841488 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.841457 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753af74e-2e36-4324-aeca-e627b40eaee5-istiod-ca-cert\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.843141 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.843122 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-istio-envoy\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.843439 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.843421 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753af74e-2e36-4324-aeca-e627b40eaee5-istio-podinfo\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.850092 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.850058 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753af74e-2e36-4324-aeca-e627b40eaee5-istio-token\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.850256 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.850238 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rmd\" (UniqueName: \"kubernetes.io/projected/753af74e-2e36-4324-aeca-e627b40eaee5-kube-api-access-v8rmd\") pod \"e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:44.887784 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:44.887754 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:45.035987 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:45.035961 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx"] Jul 29 21:18:45.037894 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:18:45.037861 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753af74e_2e36_4324_aeca_e627b40eaee5.slice/crio-0687ef1374d0dc04e811a6e3d89cdb10d0fb79a692d8bd66071c7708be719248 WatchSource:0}: Error finding container 0687ef1374d0dc04e811a6e3d89cdb10d0fb79a692d8bd66071c7708be719248: Status 404 returned error can't find the container with id 0687ef1374d0dc04e811a6e3d89cdb10d0fb79a692d8bd66071c7708be719248 Jul 29 21:18:45.040066 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:45.040008 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:18:45.040149 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:45.040120 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:18:45.040195 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:45.040166 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:18:45.222160 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:45.222119 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" event={"ID":"753af74e-2e36-4324-aeca-e627b40eaee5","Type":"ContainerStarted","Data":"4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967"} Jul 29 21:18:45.222639 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:45.222193 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" event={"ID":"753af74e-2e36-4324-aeca-e627b40eaee5","Type":"ContainerStarted","Data":"0687ef1374d0dc04e811a6e3d89cdb10d0fb79a692d8bd66071c7708be719248"} Jul 29 21:18:45.244350 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:45.244283 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" podStartSLOduration=1.244262027 podStartE2EDuration="1.244262027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:45.241917838 +0000 UTC m=+1873.365507682" watchObservedRunningTime="2026-07-29 21:18:45.244262027 +0000 UTC m=+1873.367851871" Jul 29 21:18:45.888825 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:45.888785 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:46.896953 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:46.896911 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:46.979083 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:46.979043 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:18:47.233603 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.233510 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:47.234716 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.234694 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:18:47.575082 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.575040 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95"] Jul 29 21:18:47.578572 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.578555 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" Jul 29 21:18:47.583328 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.583307 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9900cabf-dockercfg-2clzw\"" Jul 29 21:18:47.583447 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.583365 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9900cabf\"" Jul 29 21:18:47.600998 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.600972 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95"] Jul 29 21:18:47.667563 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.667531 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4skcr\" (UniqueName: \"kubernetes.io/projected/80f50d00-835b-4f99-8ee1-efea23f7325d-kube-api-access-4skcr\") pod \"payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95\" (UID: \"80f50d00-835b-4f99-8ee1-efea23f7325d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" Jul 29 21:18:47.667796 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.667605 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f50d00-835b-4f99-8ee1-efea23f7325d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95\" (UID: \"80f50d00-835b-4f99-8ee1-efea23f7325d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" Jul 29 21:18:47.704242 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.704194 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5"] Jul 29 21:18:47.711571 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.711527 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5"] Jul 29 21:18:47.711751 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.711646 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" Jul 29 21:18:47.768911 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.768852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4skcr\" (UniqueName: \"kubernetes.io/projected/80f50d00-835b-4f99-8ee1-efea23f7325d-kube-api-access-4skcr\") pod \"payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95\" (UID: \"80f50d00-835b-4f99-8ee1-efea23f7325d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" Jul 29 21:18:47.768911 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.768914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f50d00-835b-4f99-8ee1-efea23f7325d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95\" (UID: \"80f50d00-835b-4f99-8ee1-efea23f7325d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" Jul 29 21:18:47.769641 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.769621 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f50d00-835b-4f99-8ee1-efea23f7325d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95\" (UID: \"80f50d00-835b-4f99-8ee1-efea23f7325d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" Jul 29 21:18:47.779247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.779220 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4skcr\" (UniqueName: \"kubernetes.io/projected/80f50d00-835b-4f99-8ee1-efea23f7325d-kube-api-access-4skcr\") pod \"payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95\" (UID: \"80f50d00-835b-4f99-8ee1-efea23f7325d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" Jul 29 21:18:47.870098 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.869973 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4s2\" (UniqueName: \"kubernetes.io/projected/b92542b2-724c-47c7-85af-66f482293d4d-kube-api-access-hc4s2\") pod \"payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5\" (UID: \"b92542b2-724c-47c7-85af-66f482293d4d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" Jul 29 21:18:47.870256 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.870123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92542b2-724c-47c7-85af-66f482293d4d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5\" (UID: \"b92542b2-724c-47c7-85af-66f482293d4d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" Jul 29 21:18:47.889625 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.889596 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" Jul 29 21:18:47.971714 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.971671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4s2\" (UniqueName: \"kubernetes.io/projected/b92542b2-724c-47c7-85af-66f482293d4d-kube-api-access-hc4s2\") pod \"payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5\" (UID: \"b92542b2-724c-47c7-85af-66f482293d4d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" Jul 29 21:18:47.972118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.971765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92542b2-724c-47c7-85af-66f482293d4d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5\" (UID: \"b92542b2-724c-47c7-85af-66f482293d4d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" Jul 29 21:18:47.972498 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.972479 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92542b2-724c-47c7-85af-66f482293d4d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5\" (UID: \"b92542b2-724c-47c7-85af-66f482293d4d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" Jul 29 21:18:47.983635 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:47.983606 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4s2\" (UniqueName: \"kubernetes.io/projected/b92542b2-724c-47c7-85af-66f482293d4d-kube-api-access-hc4s2\") pod \"payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5\" (UID: \"b92542b2-724c-47c7-85af-66f482293d4d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" Jul 29 21:18:48.022439 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.022403 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" Jul 29 21:18:48.036221 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.036194 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95"] Jul 29 21:18:48.039070 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:18:48.039040 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f50d00_835b_4f99_8ee1_efea23f7325d.slice/crio-8662a574ae9322ce6d734f237b9d15a87ea31e3c72690cb3bdacf293c9fb7d3a WatchSource:0}: Error finding container 8662a574ae9322ce6d734f237b9d15a87ea31e3c72690cb3bdacf293c9fb7d3a: Status 404 returned error can't find the container with id 8662a574ae9322ce6d734f237b9d15a87ea31e3c72690cb3bdacf293c9fb7d3a Jul 29 21:18:48.172514 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.172485 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5"] Jul 29 21:18:48.174111 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:18:48.174084 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92542b2_724c_47c7_85af_66f482293d4d.slice/crio-c03c0cffc452d63d78f16497baa96ed449e750de298651be9dc854373bae2e2a WatchSource:0}: Error finding container c03c0cffc452d63d78f16497baa96ed449e750de298651be9dc854373bae2e2a: Status 404 returned error can't find the container with id c03c0cffc452d63d78f16497baa96ed449e750de298651be9dc854373bae2e2a Jul 29 21:18:48.240594 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.240562 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" event={"ID":"b92542b2-724c-47c7-85af-66f482293d4d","Type":"ContainerStarted","Data":"c03c0cffc452d63d78f16497baa96ed449e750de298651be9dc854373bae2e2a"} Jul 29 21:18:48.242389 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.242363 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" event={"ID":"80f50d00-835b-4f99-8ee1-efea23f7325d","Type":"ContainerStarted","Data":"48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8"} Jul 29 21:18:48.242520 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.242418 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" event={"ID":"80f50d00-835b-4f99-8ee1-efea23f7325d","Type":"ContainerStarted","Data":"8662a574ae9322ce6d734f237b9d15a87ea31e3c72690cb3bdacf293c9fb7d3a"} Jul 29 21:18:48.258534 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.258478 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" podStartSLOduration=1.258458646 podStartE2EDuration="1.258458646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:48.258239819 +0000 UTC m=+1876.381829663" watchObservedRunningTime="2026-07-29 21:18:48.258458646 +0000 UTC m=+1876.382048489" Jul 29 21:18:48.542233 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.542188 2582 patch_prober.go:28] interesting pod/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 29 21:18:48.542404 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.542249 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 29 21:18:48.750593 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.750524 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:48.881190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.881154 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbb5l\" (UniqueName: \"kubernetes.io/projected/0168312c-79e1-4a70-a3d2-0313a262d941-kube-api-access-cbb5l\") pod \"0168312c-79e1-4a70-a3d2-0313a262d941\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " Jul 29 21:18:48.881190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.881196 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-istio-data\") pod \"0168312c-79e1-4a70-a3d2-0313a262d941\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " Jul 29 21:18:48.881465 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.881224 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0168312c-79e1-4a70-a3d2-0313a262d941-istio-podinfo\") pod \"0168312c-79e1-4a70-a3d2-0313a262d941\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " Jul 29 21:18:48.881465 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.881260 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-credential-socket\") pod \"0168312c-79e1-4a70-a3d2-0313a262d941\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " Jul 29 21:18:48.881465 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.881280 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-istio-envoy\") pod \"0168312c-79e1-4a70-a3d2-0313a262d941\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " Jul 29 21:18:48.881465 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.881355 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0168312c-79e1-4a70-a3d2-0313a262d941-istio-token\") pod \"0168312c-79e1-4a70-a3d2-0313a262d941\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " Jul 29 21:18:48.881465 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.881435 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-workload-socket\") pod \"0168312c-79e1-4a70-a3d2-0313a262d941\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " Jul 29 21:18:48.881465 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.881459 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0168312c-79e1-4a70-a3d2-0313a262d941-istiod-ca-cert\") pod \"0168312c-79e1-4a70-a3d2-0313a262d941\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " Jul 29 21:18:48.881736 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.881496 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-workload-certs\") pod \"0168312c-79e1-4a70-a3d2-0313a262d941\" (UID: \"0168312c-79e1-4a70-a3d2-0313a262d941\") " Jul 29 21:18:48.881876 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.881830 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0168312c-79e1-4a70-a3d2-0313a262d941" (UID: "0168312c-79e1-4a70-a3d2-0313a262d941"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:18:48.882157 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.881891 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0168312c-79e1-4a70-a3d2-0313a262d941" (UID: "0168312c-79e1-4a70-a3d2-0313a262d941"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:18:48.882157 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.882095 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0168312c-79e1-4a70-a3d2-0313a262d941" (UID: "0168312c-79e1-4a70-a3d2-0313a262d941"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:18:48.882394 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.882368 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0168312c-79e1-4a70-a3d2-0313a262d941-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0168312c-79e1-4a70-a3d2-0313a262d941" (UID: "0168312c-79e1-4a70-a3d2-0313a262d941"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:18:48.882394 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.882373 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-istio-data" (OuterVolumeSpecName: "istio-data") pod "0168312c-79e1-4a70-a3d2-0313a262d941" (UID: "0168312c-79e1-4a70-a3d2-0313a262d941"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:18:48.883629 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.883603 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0168312c-79e1-4a70-a3d2-0313a262d941-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0168312c-79e1-4a70-a3d2-0313a262d941" (UID: "0168312c-79e1-4a70-a3d2-0313a262d941"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:18:48.883629 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.883623 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0168312c-79e1-4a70-a3d2-0313a262d941-kube-api-access-cbb5l" (OuterVolumeSpecName: "kube-api-access-cbb5l") pod "0168312c-79e1-4a70-a3d2-0313a262d941" (UID: "0168312c-79e1-4a70-a3d2-0313a262d941"). InnerVolumeSpecName "kube-api-access-cbb5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:18:48.883760 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.883715 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0168312c-79e1-4a70-a3d2-0313a262d941" (UID: "0168312c-79e1-4a70-a3d2-0313a262d941"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:18:48.884378 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.884364 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0168312c-79e1-4a70-a3d2-0313a262d941-istio-token" (OuterVolumeSpecName: "istio-token") pod "0168312c-79e1-4a70-a3d2-0313a262d941" (UID: "0168312c-79e1-4a70-a3d2-0313a262d941"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:18:48.982905 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.982863 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:18:48.982905 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.982900 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0168312c-79e1-4a70-a3d2-0313a262d941-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:18:48.982905 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.982909 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:18:48.982905 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.982917 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbb5l\" (UniqueName: \"kubernetes.io/projected/0168312c-79e1-4a70-a3d2-0313a262d941-kube-api-access-cbb5l\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:18:48.983440 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.982927 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:18:48.983440 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.982936 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0168312c-79e1-4a70-a3d2-0313a262d941-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:18:48.983440 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.982944 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:18:48.983440 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.982951 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0168312c-79e1-4a70-a3d2-0313a262d941-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:18:48.983440 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:48.982959 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0168312c-79e1-4a70-a3d2-0313a262d941-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:18:49.248734 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.248700 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" event={"ID":"b92542b2-724c-47c7-85af-66f482293d4d","Type":"ContainerStarted","Data":"33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3"} Jul 29 21:18:49.248921 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.248764 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" Jul 29 21:18:49.250496 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.250469 2582 generic.go:356] "Generic (PLEG): container finished" podID="0168312c-79e1-4a70-a3d2-0313a262d941" containerID="95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091" exitCode=0 Jul 29 21:18:49.250634 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.250507 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" Jul 29 21:18:49.250634 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.250536 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" event={"ID":"0168312c-79e1-4a70-a3d2-0313a262d941","Type":"ContainerDied","Data":"95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091"} Jul 29 21:18:49.250634 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.250586 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg" event={"ID":"0168312c-79e1-4a70-a3d2-0313a262d941","Type":"ContainerDied","Data":"8ac4e7355712c5766c113af9ff5cc501f6584c61b2e9d6fa9b0e109daa9ca453"} Jul 29 21:18:49.250634 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.250618 2582 scope.go:117] "RemoveContainer" containerID="95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091" Jul 29 21:18:49.250938 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.250917 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" Jul 29 21:18:49.260489 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.260467 2582 scope.go:117] "RemoveContainer" containerID="e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46" Jul 29 21:18:49.266773 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.266730 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" podStartSLOduration=2.266714255 podStartE2EDuration="2.266714255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:49.264274045 +0000 UTC m=+1877.387863887" watchObservedRunningTime="2026-07-29 21:18:49.266714255 +0000 UTC m=+1877.390304097" Jul 29 21:18:49.269545 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.269525 2582 scope.go:117] "RemoveContainer" containerID="95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091" Jul 29 21:18:49.269803 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:18:49.269784 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091\": container with ID starting with 95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091 not found: ID does not exist" containerID="95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091" Jul 29 21:18:49.269860 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.269812 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091"} err="failed to get container status \"95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091\": rpc error: code = NotFound desc = could not find container \"95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091\": container with ID starting with 95444265caa2fc33dc0ae981148cf2b9a0b4a87a2776e31dee79be3ace05a091 not found: ID does not exist" Jul 29 21:18:49.269860 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.269829 2582 scope.go:117] "RemoveContainer" containerID="e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46" Jul 29 21:18:49.270108 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:18:49.270089 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46\": container with ID starting with e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46 not found: ID does not exist" containerID="e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46" Jul 29 21:18:49.270161 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.270114 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46"} err="failed to get container status \"e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46\": rpc error: code = NotFound desc = could not find container \"e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46\": container with ID starting with e1bbc68840b8bcfb9b04cdbd31e48dc088650b8ef21161e071fe3c61148dbf46 not found: ID does not exist" Jul 29 21:18:49.280195 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.280171 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg"] Jul 29 21:18:49.284219 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:49.284196 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dbfaf7e5-openshift-default-84cc9c4cc6-pcghg"] Jul 29 21:18:50.492739 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:50.492706 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" path="/var/lib/kubelet/pods/0168312c-79e1-4a70-a3d2-0313a262d941/volumes" Jul 29 21:18:59.558951 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:18:59.558904 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:19:00.259548 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:00.259514 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" Jul 29 21:19:00.259729 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:00.259595 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" Jul 29 21:19:02.234864 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:02.234833 2582 patch_prober.go:28] interesting pod/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 29 21:19:02.235309 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:02.234887 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 29 21:19:02.331063 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:02.331006 2582 generic.go:356] "Generic (PLEG): container finished" podID="753af74e-2e36-4324-aeca-e627b40eaee5" containerID="4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967" exitCode=0 Jul 29 21:19:02.331236 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:02.331080 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" event={"ID":"753af74e-2e36-4324-aeca-e627b40eaee5","Type":"ContainerDied","Data":"4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967"} Jul 29 21:19:02.331474 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:02.331460 2582 scope.go:117] "RemoveContainer" containerID="4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967" Jul 29 21:19:02.332488 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:02.332458 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:19:02.332547 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:02.332530 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:19:02.332585 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:02.332560 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:19:03.343448 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:03.343410 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" event={"ID":"753af74e-2e36-4324-aeca-e627b40eaee5","Type":"ContainerStarted","Data":"60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71"} Jul 29 21:19:03.888451 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:03.888411 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:19:03.888727 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:03.888702 2582 patch_prober.go:28] interesting pod/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 29 21:19:03.888790 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:03.888768 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 29 21:19:04.888987 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:04.888945 2582 patch_prober.go:28] interesting pod/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 29 21:19:04.889504 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:04.889043 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 29 21:19:05.889250 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:05.889218 2582 patch_prober.go:28] interesting pod/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 29 21:19:05.889660 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:05.889281 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 29 21:19:06.888627 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:06.888590 2582 patch_prober.go:28] interesting pod/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 29 21:19:06.888804 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:06.888663 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 29 21:19:07.892462 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:07.892434 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:19:07.892958 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:07.892817 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:19:07.893507 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:07.893489 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:19:11.788345 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.788314 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2"] Jul 29 21:19:11.788758 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.788694 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" Jul 29 21:19:11.788758 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.788705 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" Jul 29 21:19:11.788758 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.788716 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" Jul 29 21:19:11.788758 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.788721 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" Jul 29 21:19:11.788894 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.788782 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" Jul 29 21:19:11.788894 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.788791 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0168312c-79e1-4a70-a3d2-0313a262d941" containerName="istio-proxy" Jul 29 21:19:11.791079 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.791063 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2" Jul 29 21:19:11.793796 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.793774 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:19:11.793796 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.793786 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\"" Jul 29 21:19:11.799936 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.799912 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2"] Jul 29 21:19:11.894610 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.894577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff61d878-34a9-4363-8145-560140c96da2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2\" (UID: \"ff61d878-34a9-4363-8145-560140c96da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2" Jul 29 21:19:11.995842 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.995816 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff61d878-34a9-4363-8145-560140c96da2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2\" (UID: \"ff61d878-34a9-4363-8145-560140c96da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2" Jul 29 21:19:11.996436 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:11.996417 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff61d878-34a9-4363-8145-560140c96da2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2\" (UID: \"ff61d878-34a9-4363-8145-560140c96da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2" Jul 29 21:19:12.102124 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:12.102062 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2" Jul 29 21:19:12.233071 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:12.233042 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2"] Jul 29 21:19:12.391078 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:12.390983 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2" event={"ID":"ff61d878-34a9-4363-8145-560140c96da2","Type":"ContainerStarted","Data":"b09590d1d42fd2e202a6051f11582fb83dabb46e7964a995eea35ab0f0ca16a2"} Jul 29 21:19:13.397892 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:13.397855 2582 generic.go:356] "Generic (PLEG): container finished" podID="ff61d878-34a9-4363-8145-560140c96da2" containerID="b2e61e6ff96dbbe22d4ab75535710e839fd9337855c373d0fe89566cf50d0c41" exitCode=0 Jul 29 21:19:13.398311 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:13.397898 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2" event={"ID":"ff61d878-34a9-4363-8145-560140c96da2","Type":"ContainerDied","Data":"b2e61e6ff96dbbe22d4ab75535710e839fd9337855c373d0fe89566cf50d0c41"} Jul 29 21:19:14.537105 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:14.537078 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2" Jul 29 21:19:14.620666 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:14.620640 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff61d878-34a9-4363-8145-560140c96da2-maas-api-service-ca\") pod \"ff61d878-34a9-4363-8145-560140c96da2\" (UID: \"ff61d878-34a9-4363-8145-560140c96da2\") " Jul 29 21:19:14.620991 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:14.620968 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff61d878-34a9-4363-8145-560140c96da2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff61d878-34a9-4363-8145-560140c96da2" (UID: "ff61d878-34a9-4363-8145-560140c96da2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:19:14.721456 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:14.721384 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff61d878-34a9-4363-8145-560140c96da2-maas-api-service-ca\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:15.409767 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:15.409737 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2" Jul 29 21:19:15.409940 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:15.409789 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2" event={"ID":"ff61d878-34a9-4363-8145-560140c96da2","Type":"ContainerDied","Data":"b09590d1d42fd2e202a6051f11582fb83dabb46e7964a995eea35ab0f0ca16a2"} Jul 29 21:19:15.409940 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:15.409841 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09590d1d42fd2e202a6051f11582fb83dabb46e7964a995eea35ab0f0ca16a2" Jul 29 21:19:17.604198 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.604163 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5"] Jul 29 21:19:17.604764 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.604413 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" podUID="b92542b2-724c-47c7-85af-66f482293d4d" containerName="payload-processing" containerID="cri-o://33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3" gracePeriod=30 Jul 29 21:19:17.610825 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.610798 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95"] Jul 29 21:19:17.611232 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.611192 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" podUID="80f50d00-835b-4f99-8ee1-efea23f7325d" containerName="payload-pre-processing" containerID="cri-o://48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8" gracePeriod=30 Jul 29 21:19:17.780524 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.780496 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2"] Jul 29 21:19:17.782966 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.782645 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9900cabf-8tbp2"] Jul 29 21:19:17.893929 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.893855 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" Jul 29 21:19:17.900046 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.898878 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" Jul 29 21:19:17.931088 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.931056 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx"] Jul 29 21:19:17.931322 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.931297 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" containerID="cri-o://60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71" gracePeriod=30 Jul 29 21:19:17.946039 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.946002 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92542b2-724c-47c7-85af-66f482293d4d-plugins-config-volume\") pod \"b92542b2-724c-47c7-85af-66f482293d4d\" (UID: \"b92542b2-724c-47c7-85af-66f482293d4d\") " Jul 29 21:19:17.946148 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.946129 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4skcr\" (UniqueName: \"kubernetes.io/projected/80f50d00-835b-4f99-8ee1-efea23f7325d-kube-api-access-4skcr\") pod \"80f50d00-835b-4f99-8ee1-efea23f7325d\" (UID: \"80f50d00-835b-4f99-8ee1-efea23f7325d\") " Jul 29 21:19:17.946191 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.946161 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f50d00-835b-4f99-8ee1-efea23f7325d-plugins-config-volume\") pod \"80f50d00-835b-4f99-8ee1-efea23f7325d\" (UID: \"80f50d00-835b-4f99-8ee1-efea23f7325d\") " Jul 29 21:19:17.946244 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.946193 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc4s2\" (UniqueName: \"kubernetes.io/projected/b92542b2-724c-47c7-85af-66f482293d4d-kube-api-access-hc4s2\") pod \"b92542b2-724c-47c7-85af-66f482293d4d\" (UID: \"b92542b2-724c-47c7-85af-66f482293d4d\") " Jul 29 21:19:17.946476 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.946446 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92542b2-724c-47c7-85af-66f482293d4d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b92542b2-724c-47c7-85af-66f482293d4d" (UID: "b92542b2-724c-47c7-85af-66f482293d4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:19:17.946573 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.946536 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f50d00-835b-4f99-8ee1-efea23f7325d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80f50d00-835b-4f99-8ee1-efea23f7325d" (UID: "80f50d00-835b-4f99-8ee1-efea23f7325d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:19:17.948294 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.948265 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f50d00-835b-4f99-8ee1-efea23f7325d-kube-api-access-4skcr" (OuterVolumeSpecName: "kube-api-access-4skcr") pod "80f50d00-835b-4f99-8ee1-efea23f7325d" (UID: "80f50d00-835b-4f99-8ee1-efea23f7325d"). InnerVolumeSpecName "kube-api-access-4skcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:19:17.948400 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:17.948343 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92542b2-724c-47c7-85af-66f482293d4d-kube-api-access-hc4s2" (OuterVolumeSpecName: "kube-api-access-hc4s2") pod "b92542b2-724c-47c7-85af-66f482293d4d" (UID: "b92542b2-724c-47c7-85af-66f482293d4d"). InnerVolumeSpecName "kube-api-access-hc4s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:19:18.047330 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.047299 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4skcr\" (UniqueName: \"kubernetes.io/projected/80f50d00-835b-4f99-8ee1-efea23f7325d-kube-api-access-4skcr\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:18.047330 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.047327 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f50d00-835b-4f99-8ee1-efea23f7325d-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:18.047330 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.047339 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc4s2\" (UniqueName: \"kubernetes.io/projected/b92542b2-724c-47c7-85af-66f482293d4d-kube-api-access-hc4s2\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:18.047564 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.047351 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92542b2-724c-47c7-85af-66f482293d4d-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:18.432600 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.432569 2582 generic.go:356] "Generic (PLEG): container finished" podID="b92542b2-724c-47c7-85af-66f482293d4d" containerID="33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3" exitCode=0 Jul 29 21:19:18.432811 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.432624 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" Jul 29 21:19:18.432811 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.432645 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" event={"ID":"b92542b2-724c-47c7-85af-66f482293d4d","Type":"ContainerDied","Data":"33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3"} Jul 29 21:19:18.432811 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.432713 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5" event={"ID":"b92542b2-724c-47c7-85af-66f482293d4d","Type":"ContainerDied","Data":"c03c0cffc452d63d78f16497baa96ed449e750de298651be9dc854373bae2e2a"} Jul 29 21:19:18.432811 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.432744 2582 scope.go:117] "RemoveContainer" containerID="33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3" Jul 29 21:19:18.433995 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.433974 2582 generic.go:356] "Generic (PLEG): container finished" podID="80f50d00-835b-4f99-8ee1-efea23f7325d" containerID="48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8" exitCode=0 Jul 29 21:19:18.434136 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.434036 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" Jul 29 21:19:18.434136 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.434062 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" event={"ID":"80f50d00-835b-4f99-8ee1-efea23f7325d","Type":"ContainerDied","Data":"48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8"} Jul 29 21:19:18.434136 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.434117 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95" event={"ID":"80f50d00-835b-4f99-8ee1-efea23f7325d","Type":"ContainerDied","Data":"8662a574ae9322ce6d734f237b9d15a87ea31e3c72690cb3bdacf293c9fb7d3a"} Jul 29 21:19:18.442014 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.441993 2582 scope.go:117] "RemoveContainer" containerID="33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3" Jul 29 21:19:18.442391 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:19:18.442364 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3\": container with ID starting with 33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3 not found: ID does not exist" containerID="33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3" Jul 29 21:19:18.442468 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.442392 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3"} err="failed to get container status \"33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3\": rpc error: code = NotFound desc = could not find container \"33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3\": container with ID starting with 33bf76189be533e24c488510079354a1178c04ebfe4c6e01f0da4271b587cab3 not found: ID does not exist" Jul 29 21:19:18.442468 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.442409 2582 scope.go:117] "RemoveContainer" containerID="48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8" Jul 29 21:19:18.450864 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.450845 2582 scope.go:117] "RemoveContainer" containerID="48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8" Jul 29 21:19:18.451157 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:19:18.451139 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8\": container with ID starting with 48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8 not found: ID does not exist" containerID="48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8" Jul 29 21:19:18.451221 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.451166 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8"} err="failed to get container status \"48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8\": rpc error: code = NotFound desc = could not find container \"48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8\": container with ID starting with 48588cae2b2465b89d6988ae3b6d2ffcc482411bf9f1e9acdbd69a60d24fecc8 not found: ID does not exist" Jul 29 21:19:18.459735 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.459714 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95"] Jul 29 21:19:18.464148 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.464130 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9900cabf-598f5ff4dd-5rt95"] Jul 29 21:19:18.476594 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.476568 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5"] Jul 29 21:19:18.479892 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.479868 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9900cabf-7f5664bc97-6mgf5"] Jul 29 21:19:18.493423 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.493392 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f50d00-835b-4f99-8ee1-efea23f7325d" path="/var/lib/kubelet/pods/80f50d00-835b-4f99-8ee1-efea23f7325d/volumes" Jul 29 21:19:18.493876 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.493857 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b92542b2-724c-47c7-85af-66f482293d4d" path="/var/lib/kubelet/pods/b92542b2-724c-47c7-85af-66f482293d4d/volumes" Jul 29 21:19:18.494310 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.494291 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff61d878-34a9-4363-8145-560140c96da2" path="/var/lib/kubelet/pods/ff61d878-34a9-4363-8145-560140c96da2/volumes" Jul 29 21:19:18.727951 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.727864 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f"] Jul 29 21:19:18.728408 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.728376 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f50d00-835b-4f99-8ee1-efea23f7325d" containerName="payload-pre-processing" Jul 29 21:19:18.728408 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.728395 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f50d00-835b-4f99-8ee1-efea23f7325d" containerName="payload-pre-processing" Jul 29 21:19:18.728524 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.728426 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff61d878-34a9-4363-8145-560140c96da2" containerName="revoke-keys" Jul 29 21:19:18.728524 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.728436 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff61d878-34a9-4363-8145-560140c96da2" containerName="revoke-keys" Jul 29 21:19:18.728524 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.728451 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b92542b2-724c-47c7-85af-66f482293d4d" containerName="payload-processing" Jul 29 21:19:18.728524 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.728460 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b92542b2-724c-47c7-85af-66f482293d4d" containerName="payload-processing" Jul 29 21:19:18.728720 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.728557 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b92542b2-724c-47c7-85af-66f482293d4d" containerName="payload-processing" Jul 29 21:19:18.728720 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.728572 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f50d00-835b-4f99-8ee1-efea23f7325d" containerName="payload-pre-processing" Jul 29 21:19:18.728720 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.728583 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff61d878-34a9-4363-8145-560140c96da2" containerName="revoke-keys" Jul 29 21:19:18.731686 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.731664 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.734774 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.734758 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e74c91e4-openshift-default-dockercfg-rm6lf\"" Jul 29 21:19:18.746004 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.745984 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f"] Jul 29 21:19:18.854287 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.854251 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-workload-certs\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.854491 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.854295 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz8f9\" (UniqueName: \"kubernetes.io/projected/778953bd-eed8-4ad5-ac7d-d38f0e504830-kube-api-access-vz8f9\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.854491 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.854328 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-envoy\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.854491 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.854425 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-credential-socket\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.854491 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.854482 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-podinfo\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.854716 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.854544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-token\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.854716 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.854575 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-workload-socket\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.854716 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.854648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-data\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.854716 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.854673 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778953bd-eed8-4ad5-ac7d-d38f0e504830-istiod-ca-cert\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.955606 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.955570 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-data\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.955606 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.955608 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778953bd-eed8-4ad5-ac7d-d38f0e504830-istiod-ca-cert\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.955860 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.955636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-workload-certs\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.955860 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.955656 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8f9\" (UniqueName: \"kubernetes.io/projected/778953bd-eed8-4ad5-ac7d-d38f0e504830-kube-api-access-vz8f9\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.955860 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.955675 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-envoy\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.955860 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.955699 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-credential-socket\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.955860 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.955738 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-podinfo\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.955860 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.955788 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-token\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.955860 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.955812 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-workload-socket\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.956243 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.956058 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-data\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.956243 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.956068 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-workload-certs\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.956243 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.956166 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-workload-socket\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.956440 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.956421 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-credential-socket\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.956525 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.956505 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778953bd-eed8-4ad5-ac7d-d38f0e504830-istiod-ca-cert\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.958173 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.958151 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-envoy\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.958284 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.958261 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-podinfo\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.968572 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.968544 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz8f9\" (UniqueName: \"kubernetes.io/projected/778953bd-eed8-4ad5-ac7d-d38f0e504830-kube-api-access-vz8f9\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:18.971413 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:18.971393 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-token\") pod \"e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:19.041237 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:19.041207 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:19.184008 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:19.183982 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f"] Jul 29 21:19:19.185144 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:19:19.185106 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778953bd_eed8_4ad5_ac7d_d38f0e504830.slice/crio-6b596ad6053bb2ece75526a53334f3fe7344eea5751972735a3c1e1d21b23b9e WatchSource:0}: Error finding container 6b596ad6053bb2ece75526a53334f3fe7344eea5751972735a3c1e1d21b23b9e: Status 404 returned error can't find the container with id 6b596ad6053bb2ece75526a53334f3fe7344eea5751972735a3c1e1d21b23b9e Jul 29 21:19:19.187346 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:19.187311 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:19:19.187447 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:19.187377 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:19:19.187447 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:19.187417 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:19:19.441887 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:19.441773 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" event={"ID":"778953bd-eed8-4ad5-ac7d-d38f0e504830","Type":"ContainerStarted","Data":"4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93"} Jul 29 21:19:19.441887 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:19.441840 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" event={"ID":"778953bd-eed8-4ad5-ac7d-d38f0e504830","Type":"ContainerStarted","Data":"6b596ad6053bb2ece75526a53334f3fe7344eea5751972735a3c1e1d21b23b9e"} Jul 29 21:19:19.463913 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:19.463853 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" podStartSLOduration=1.463829515 podStartE2EDuration="1.463829515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:19:19.461957218 +0000 UTC m=+1907.585547061" watchObservedRunningTime="2026-07-29 21:19:19.463829515 +0000 UTC m=+1907.587419359" Jul 29 21:19:20.042032 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:20.041998 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:20.047243 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:20.047217 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:20.449408 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:20.449326 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:20.450531 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:20.450510 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:21.757565 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:21.757530 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:19:22.678919 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.678883 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6"] Jul 29 21:19:22.681727 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.681707 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" Jul 29 21:19:22.684418 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.684401 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e74c91e4-dockercfg-76mrl\"" Jul 29 21:19:22.684691 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.684672 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e74c91e4\"" Jul 29 21:19:22.695904 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.695871 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6"] Jul 29 21:19:22.700371 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.700338 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg"] Jul 29 21:19:22.702888 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.702867 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" Jul 29 21:19:22.719142 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.719109 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg"] Jul 29 21:19:22.792867 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.792821 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd1f3f6-550f-4bac-960b-a61c0656ba1e-plugins-config-volume\") pod \"payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg\" (UID: \"ecd1f3f6-550f-4bac-960b-a61c0656ba1e\") " pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" Jul 29 21:19:22.793378 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.792935 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qh6\" (UniqueName: \"kubernetes.io/projected/eb2a6efd-ce05-4b81-96c7-637db360b341-kube-api-access-t2qh6\") pod \"payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6\" (UID: \"eb2a6efd-ce05-4b81-96c7-637db360b341\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" Jul 29 21:19:22.793378 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.793079 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzw6\" (UniqueName: \"kubernetes.io/projected/ecd1f3f6-550f-4bac-960b-a61c0656ba1e-kube-api-access-9kzw6\") pod \"payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg\" (UID: \"ecd1f3f6-550f-4bac-960b-a61c0656ba1e\") " pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" Jul 29 21:19:22.793378 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.793145 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2a6efd-ce05-4b81-96c7-637db360b341-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6\" (UID: \"eb2a6efd-ce05-4b81-96c7-637db360b341\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" Jul 29 21:19:22.894103 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.894055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2a6efd-ce05-4b81-96c7-637db360b341-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6\" (UID: \"eb2a6efd-ce05-4b81-96c7-637db360b341\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" Jul 29 21:19:22.894301 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.894135 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd1f3f6-550f-4bac-960b-a61c0656ba1e-plugins-config-volume\") pod \"payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg\" (UID: \"ecd1f3f6-550f-4bac-960b-a61c0656ba1e\") " pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" Jul 29 21:19:22.894301 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.894181 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qh6\" (UniqueName: \"kubernetes.io/projected/eb2a6efd-ce05-4b81-96c7-637db360b341-kube-api-access-t2qh6\") pod \"payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6\" (UID: \"eb2a6efd-ce05-4b81-96c7-637db360b341\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" Jul 29 21:19:22.894301 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.894266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzw6\" (UniqueName: \"kubernetes.io/projected/ecd1f3f6-550f-4bac-960b-a61c0656ba1e-kube-api-access-9kzw6\") pod \"payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg\" (UID: \"ecd1f3f6-550f-4bac-960b-a61c0656ba1e\") " pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" Jul 29 21:19:22.895132 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.895095 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2a6efd-ce05-4b81-96c7-637db360b341-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6\" (UID: \"eb2a6efd-ce05-4b81-96c7-637db360b341\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" Jul 29 21:19:22.895405 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.895380 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd1f3f6-550f-4bac-960b-a61c0656ba1e-plugins-config-volume\") pod \"payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg\" (UID: \"ecd1f3f6-550f-4bac-960b-a61c0656ba1e\") " pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" Jul 29 21:19:22.896106 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.896078 2582 patch_prober.go:28] interesting pod/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 21:19:22.896248 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.896136 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 21:19:22.903476 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.903451 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzw6\" (UniqueName: \"kubernetes.io/projected/ecd1f3f6-550f-4bac-960b-a61c0656ba1e-kube-api-access-9kzw6\") pod \"payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg\" (UID: \"ecd1f3f6-550f-4bac-960b-a61c0656ba1e\") " pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" Jul 29 21:19:22.903604 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.903515 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qh6\" (UniqueName: \"kubernetes.io/projected/eb2a6efd-ce05-4b81-96c7-637db360b341-kube-api-access-t2qh6\") pod \"payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6\" (UID: \"eb2a6efd-ce05-4b81-96c7-637db360b341\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" Jul 29 21:19:22.994116 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:22.994065 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" Jul 29 21:19:23.013613 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.013585 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" Jul 29 21:19:23.156783 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.156735 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6"] Jul 29 21:19:23.159135 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:19:23.159047 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2a6efd_ce05_4b81_96c7_637db360b341.slice/crio-cfc7d525a98a91e97eee09e579e4386236920118d5710632f088cf50c99f2c20 WatchSource:0}: Error finding container cfc7d525a98a91e97eee09e579e4386236920118d5710632f088cf50c99f2c20: Status 404 returned error can't find the container with id cfc7d525a98a91e97eee09e579e4386236920118d5710632f088cf50c99f2c20 Jul 29 21:19:23.208672 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.208642 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg"] Jul 29 21:19:23.213352 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:19:23.213326 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd1f3f6_550f_4bac_960b_a61c0656ba1e.slice/crio-373c30ac12eef09ccca832509a562b6f0670165e9a6a0af3adeac19a763ba039 WatchSource:0}: Error finding container 373c30ac12eef09ccca832509a562b6f0670165e9a6a0af3adeac19a763ba039: Status 404 returned error can't find the container with id 373c30ac12eef09ccca832509a562b6f0670165e9a6a0af3adeac19a763ba039 Jul 29 21:19:23.219393 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.219370 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:19:23.301118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301086 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8rmd\" (UniqueName: \"kubernetes.io/projected/753af74e-2e36-4324-aeca-e627b40eaee5-kube-api-access-v8rmd\") pod \"753af74e-2e36-4324-aeca-e627b40eaee5\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " Jul 29 21:19:23.301305 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301150 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-istio-data\") pod \"753af74e-2e36-4324-aeca-e627b40eaee5\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " Jul 29 21:19:23.301305 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301183 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-workload-certs\") pod \"753af74e-2e36-4324-aeca-e627b40eaee5\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " Jul 29 21:19:23.301305 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301268 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-istio-envoy\") pod \"753af74e-2e36-4324-aeca-e627b40eaee5\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " Jul 29 21:19:23.301305 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301302 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-workload-socket\") pod \"753af74e-2e36-4324-aeca-e627b40eaee5\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " Jul 29 21:19:23.301519 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301354 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753af74e-2e36-4324-aeca-e627b40eaee5-istio-token\") pod \"753af74e-2e36-4324-aeca-e627b40eaee5\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " Jul 29 21:19:23.301519 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301396 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753af74e-2e36-4324-aeca-e627b40eaee5-istiod-ca-cert\") pod \"753af74e-2e36-4324-aeca-e627b40eaee5\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " Jul 29 21:19:23.301519 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301423 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-credential-socket\") pod \"753af74e-2e36-4324-aeca-e627b40eaee5\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " Jul 29 21:19:23.301519 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301459 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753af74e-2e36-4324-aeca-e627b40eaee5-istio-podinfo\") pod \"753af74e-2e36-4324-aeca-e627b40eaee5\" (UID: \"753af74e-2e36-4324-aeca-e627b40eaee5\") " Jul 29 21:19:23.301838 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301457 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "753af74e-2e36-4324-aeca-e627b40eaee5" (UID: "753af74e-2e36-4324-aeca-e627b40eaee5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:19:23.301838 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301770 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/753af74e-2e36-4324-aeca-e627b40eaee5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "753af74e-2e36-4324-aeca-e627b40eaee5" (UID: "753af74e-2e36-4324-aeca-e627b40eaee5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:19:23.301838 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301811 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "753af74e-2e36-4324-aeca-e627b40eaee5" (UID: "753af74e-2e36-4324-aeca-e627b40eaee5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:19:23.302178 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.301828 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "753af74e-2e36-4324-aeca-e627b40eaee5" (UID: "753af74e-2e36-4324-aeca-e627b40eaee5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:19:23.303122 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.303094 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-istio-data" (OuterVolumeSpecName: "istio-data") pod "753af74e-2e36-4324-aeca-e627b40eaee5" (UID: "753af74e-2e36-4324-aeca-e627b40eaee5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:19:23.304376 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.304346 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "753af74e-2e36-4324-aeca-e627b40eaee5" (UID: "753af74e-2e36-4324-aeca-e627b40eaee5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:19:23.304559 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.304502 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753af74e-2e36-4324-aeca-e627b40eaee5-istio-token" (OuterVolumeSpecName: "istio-token") pod "753af74e-2e36-4324-aeca-e627b40eaee5" (UID: "753af74e-2e36-4324-aeca-e627b40eaee5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:19:23.304559 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.304534 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753af74e-2e36-4324-aeca-e627b40eaee5-kube-api-access-v8rmd" (OuterVolumeSpecName: "kube-api-access-v8rmd") pod "753af74e-2e36-4324-aeca-e627b40eaee5" (UID: "753af74e-2e36-4324-aeca-e627b40eaee5"). InnerVolumeSpecName "kube-api-access-v8rmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:19:23.304696 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.304591 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/753af74e-2e36-4324-aeca-e627b40eaee5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "753af74e-2e36-4324-aeca-e627b40eaee5" (UID: "753af74e-2e36-4324-aeca-e627b40eaee5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:19:23.403074 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.403003 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753af74e-2e36-4324-aeca-e627b40eaee5-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:23.403074 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.403063 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:23.403074 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.403080 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753af74e-2e36-4324-aeca-e627b40eaee5-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:23.403373 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.403094 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8rmd\" (UniqueName: \"kubernetes.io/projected/753af74e-2e36-4324-aeca-e627b40eaee5-kube-api-access-v8rmd\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:23.403373 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.403107 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:23.403373 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.403119 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:23.403373 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.403130 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:23.403373 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.403142 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753af74e-2e36-4324-aeca-e627b40eaee5-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:23.403373 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.403154 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753af74e-2e36-4324-aeca-e627b40eaee5-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:23.466288 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.466196 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" event={"ID":"ecd1f3f6-550f-4bac-960b-a61c0656ba1e","Type":"ContainerStarted","Data":"f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1"} Jul 29 21:19:23.466288 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.466249 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" event={"ID":"ecd1f3f6-550f-4bac-960b-a61c0656ba1e","Type":"ContainerStarted","Data":"373c30ac12eef09ccca832509a562b6f0670165e9a6a0af3adeac19a763ba039"} Jul 29 21:19:23.466288 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.466288 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" Jul 29 21:19:23.467879 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.467833 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" event={"ID":"eb2a6efd-ce05-4b81-96c7-637db360b341","Type":"ContainerStarted","Data":"c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa"} Jul 29 21:19:23.468018 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.467905 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" event={"ID":"eb2a6efd-ce05-4b81-96c7-637db360b341","Type":"ContainerStarted","Data":"cfc7d525a98a91e97eee09e579e4386236920118d5710632f088cf50c99f2c20"} Jul 29 21:19:23.468018 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.467934 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" Jul 29 21:19:23.469660 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.469634 2582 generic.go:356] "Generic (PLEG): container finished" podID="753af74e-2e36-4324-aeca-e627b40eaee5" containerID="60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71" exitCode=0 Jul 29 21:19:23.469784 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.469673 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" Jul 29 21:19:23.469784 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.469669 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" event={"ID":"753af74e-2e36-4324-aeca-e627b40eaee5","Type":"ContainerDied","Data":"60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71"} Jul 29 21:19:23.469784 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.469782 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx" event={"ID":"753af74e-2e36-4324-aeca-e627b40eaee5","Type":"ContainerDied","Data":"0687ef1374d0dc04e811a6e3d89cdb10d0fb79a692d8bd66071c7708be719248"} Jul 29 21:19:23.469949 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.469804 2582 scope.go:117] "RemoveContainer" containerID="60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71" Jul 29 21:19:23.479356 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.479335 2582 scope.go:117] "RemoveContainer" containerID="4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967" Jul 29 21:19:23.482744 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.482703 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" podStartSLOduration=1.4826892090000001 podStartE2EDuration="1.482689209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:19:23.48144631 +0000 UTC m=+1911.605036153" watchObservedRunningTime="2026-07-29 21:19:23.482689209 +0000 UTC m=+1911.606279065" Jul 29 21:19:23.488681 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.488662 2582 scope.go:117] "RemoveContainer" containerID="60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71" Jul 29 21:19:23.488949 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:19:23.488923 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71\": container with ID starting with 60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71 not found: ID does not exist" containerID="60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71" Jul 29 21:19:23.488993 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.488959 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71"} err="failed to get container status \"60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71\": rpc error: code = NotFound desc = could not find container \"60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71\": container with ID starting with 60a971d41ae3242743475bc98664495536b453e7fb519120343f772e6d56ea71 not found: ID does not exist" Jul 29 21:19:23.488993 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.488976 2582 scope.go:117] "RemoveContainer" containerID="4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967" Jul 29 21:19:23.489216 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:19:23.489200 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967\": container with ID starting with 4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967 not found: ID does not exist" containerID="4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967" Jul 29 21:19:23.489299 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.489223 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967"} err="failed to get container status \"4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967\": rpc error: code = NotFound desc = could not find container \"4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967\": container with ID starting with 4d526052a0bf6de9b329f32c3db8aa210464051e6b96ec5c7ff4a3dfbabdf967 not found: ID does not exist" Jul 29 21:19:23.496846 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.496815 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx"] Jul 29 21:19:23.499803 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.499781 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-9900cabf-gw-openshift-default-6fd4d5c756-7brvx"] Jul 29 21:19:23.514627 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:23.514575 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" podStartSLOduration=1.514555936 podStartE2EDuration="1.514555936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:19:23.512473436 +0000 UTC m=+1911.636063280" watchObservedRunningTime="2026-07-29 21:19:23.514555936 +0000 UTC m=+1911.638145783" Jul 29 21:19:24.493529 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:24.493487 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" path="/var/lib/kubelet/pods/753af74e-2e36-4324-aeca-e627b40eaee5/volumes" Jul 29 21:19:33.346280 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:33.346246 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:19:34.485229 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:34.485200 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" Jul 29 21:19:34.485678 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:34.485499 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" Jul 29 21:19:35.701869 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:35.701492 2582 patch_prober.go:28] interesting pod/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:43154->10.134.0.57:15021: read: connection reset by peer" start-of-body= Jul 29 21:19:35.701869 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:35.701546 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:43154->10.134.0.57:15021: read: connection reset by peer" Jul 29 21:19:36.560563 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:36.560524 2582 generic.go:356] "Generic (PLEG): container finished" podID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerID="4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93" exitCode=0 Jul 29 21:19:36.560563 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:36.560552 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" event={"ID":"778953bd-eed8-4ad5-ac7d-d38f0e504830","Type":"ContainerDied","Data":"4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93"} Jul 29 21:19:36.560994 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:36.560978 2582 scope.go:117] "RemoveContainer" containerID="4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93" Jul 29 21:19:36.562041 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:36.561995 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:19:36.562146 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:36.562102 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:19:36.562210 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:36.562155 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:19:37.567930 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:37.567898 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" event={"ID":"778953bd-eed8-4ad5-ac7d-d38f0e504830","Type":"ContainerStarted","Data":"858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e"} Jul 29 21:19:38.042440 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:38.042403 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:38.042643 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:38.042609 2582 patch_prober.go:28] interesting pod/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 29 21:19:38.042692 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:38.042660 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 29 21:19:39.042239 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:39.042191 2582 patch_prober.go:28] interesting pod/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 29 21:19:39.042680 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:39.042266 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 29 21:19:40.042396 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:40.042360 2582 patch_prober.go:28] interesting pod/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 29 21:19:40.042814 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:40.042420 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 29 21:19:41.042387 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:41.042346 2582 patch_prober.go:28] interesting pod/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 29 21:19:41.042572 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:41.042424 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 29 21:19:42.041835 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:42.041795 2582 patch_prober.go:28] interesting pod/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 29 21:19:42.042063 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:42.041851 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 29 21:19:43.044798 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:43.044768 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:43.045293 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:43.045069 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:43.045856 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:43.045836 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:19:51.882003 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:51.881967 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m"] Jul 29 21:19:51.882424 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:51.882364 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" Jul 29 21:19:51.882424 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:51.882377 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" Jul 29 21:19:51.882502 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:51.882446 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" Jul 29 21:19:51.885727 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:51.885711 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m" Jul 29 21:19:51.888300 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:51.888279 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:19:51.888484 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:51.888470 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\"" Jul 29 21:19:51.894118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:51.894097 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m"] Jul 29 21:19:51.965997 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:51.965955 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e74c91e4-8759m\" (UID: \"3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m" Jul 29 21:19:52.066636 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:52.066587 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e74c91e4-8759m\" (UID: \"3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m" Jul 29 21:19:52.067307 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:52.067281 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e74c91e4-8759m\" (UID: \"3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m" Jul 29 21:19:52.197731 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:52.197638 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m" Jul 29 21:19:52.341802 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:52.341771 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m"] Jul 29 21:19:52.342732 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:19:52.342706 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed0548b_ebe5_4ee5_bbcd_3edd6aaceeaa.slice/crio-a86e37e1282c7ffd325ff3dd8658092890ed1e74655a7eecf94ffd13166d25d0 WatchSource:0}: Error finding container a86e37e1282c7ffd325ff3dd8658092890ed1e74655a7eecf94ffd13166d25d0: Status 404 returned error can't find the container with id a86e37e1282c7ffd325ff3dd8658092890ed1e74655a7eecf94ffd13166d25d0 Jul 29 21:19:52.670245 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:52.670202 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m" event={"ID":"3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa","Type":"ContainerStarted","Data":"a0a6770a46e0a0eb285a959557af145da94c7e0658d56bb9a76b77d39524ab98"} Jul 29 21:19:52.670447 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:52.670281 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m" event={"ID":"3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa","Type":"ContainerStarted","Data":"a86e37e1282c7ffd325ff3dd8658092890ed1e74655a7eecf94ffd13166d25d0"} Jul 29 21:19:52.687617 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:52.687561 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m" podStartSLOduration=1.68754677 podStartE2EDuration="1.68754677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:19:52.684735672 +0000 UTC m=+1940.808325516" watchObservedRunningTime="2026-07-29 21:19:52.68754677 +0000 UTC m=+1940.811136612" Jul 29 21:19:53.677554 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:53.677521 2582 generic.go:356] "Generic (PLEG): container finished" podID="3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa" containerID="a0a6770a46e0a0eb285a959557af145da94c7e0658d56bb9a76b77d39524ab98" exitCode=0 Jul 29 21:19:53.677554 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:53.677560 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m" event={"ID":"3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa","Type":"ContainerDied","Data":"a0a6770a46e0a0eb285a959557af145da94c7e0658d56bb9a76b77d39524ab98"} Jul 29 21:19:54.818737 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:54.818711 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m" Jul 29 21:19:54.890947 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:54.890916 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa-maas-api-service-ca\") pod \"3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa\" (UID: \"3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa\") " Jul 29 21:19:54.891278 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:54.891254 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa" (UID: "3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:19:54.992569 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:54.992530 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa-maas-api-service-ca\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:19:55.689179 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:55.689147 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m" Jul 29 21:19:55.689371 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:55.689189 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m" event={"ID":"3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa","Type":"ContainerDied","Data":"a86e37e1282c7ffd325ff3dd8658092890ed1e74655a7eecf94ffd13166d25d0"} Jul 29 21:19:55.689371 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:19:55.689243 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a86e37e1282c7ffd325ff3dd8658092890ed1e74655a7eecf94ffd13166d25d0" Jul 29 21:20:02.758939 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:02.758904 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg"] Jul 29 21:20:02.759462 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:02.759246 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" podUID="ecd1f3f6-550f-4bac-960b-a61c0656ba1e" containerName="payload-processing" containerID="cri-o://f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1" gracePeriod=30 Jul 29 21:20:02.763649 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:02.763625 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6"] Jul 29 21:20:02.764178 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:02.764125 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" podUID="eb2a6efd-ce05-4b81-96c7-637db360b341" containerName="payload-pre-processing" containerID="cri-o://c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa" gracePeriod=30 Jul 29 21:20:02.896117 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:02.896061 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m"] Jul 29 21:20:02.903167 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:02.903131 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e74c91e4-8759m"] Jul 29 21:20:03.024800 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.024736 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" Jul 29 21:20:03.057986 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.057954 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" Jul 29 21:20:03.065712 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.065691 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kzw6\" (UniqueName: \"kubernetes.io/projected/ecd1f3f6-550f-4bac-960b-a61c0656ba1e-kube-api-access-9kzw6\") pod \"ecd1f3f6-550f-4bac-960b-a61c0656ba1e\" (UID: \"ecd1f3f6-550f-4bac-960b-a61c0656ba1e\") " Jul 29 21:20:03.065848 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.065831 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd1f3f6-550f-4bac-960b-a61c0656ba1e-plugins-config-volume\") pod \"ecd1f3f6-550f-4bac-960b-a61c0656ba1e\" (UID: \"ecd1f3f6-550f-4bac-960b-a61c0656ba1e\") " Jul 29 21:20:03.066247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.066222 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd1f3f6-550f-4bac-960b-a61c0656ba1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecd1f3f6-550f-4bac-960b-a61c0656ba1e" (UID: "ecd1f3f6-550f-4bac-960b-a61c0656ba1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:20:03.067961 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.067938 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd1f3f6-550f-4bac-960b-a61c0656ba1e-kube-api-access-9kzw6" (OuterVolumeSpecName: "kube-api-access-9kzw6") pod "ecd1f3f6-550f-4bac-960b-a61c0656ba1e" (UID: "ecd1f3f6-550f-4bac-960b-a61c0656ba1e"). InnerVolumeSpecName "kube-api-access-9kzw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:20:03.167403 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.167363 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2a6efd-ce05-4b81-96c7-637db360b341-plugins-config-volume\") pod \"eb2a6efd-ce05-4b81-96c7-637db360b341\" (UID: \"eb2a6efd-ce05-4b81-96c7-637db360b341\") " Jul 29 21:20:03.167573 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.167420 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2qh6\" (UniqueName: \"kubernetes.io/projected/eb2a6efd-ce05-4b81-96c7-637db360b341-kube-api-access-t2qh6\") pod \"eb2a6efd-ce05-4b81-96c7-637db360b341\" (UID: \"eb2a6efd-ce05-4b81-96c7-637db360b341\") " Jul 29 21:20:03.167637 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.167604 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd1f3f6-550f-4bac-960b-a61c0656ba1e-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:03.167637 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.167615 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kzw6\" (UniqueName: \"kubernetes.io/projected/ecd1f3f6-550f-4bac-960b-a61c0656ba1e-kube-api-access-9kzw6\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:03.167776 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.167752 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2a6efd-ce05-4b81-96c7-637db360b341-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb2a6efd-ce05-4b81-96c7-637db360b341" (UID: "eb2a6efd-ce05-4b81-96c7-637db360b341"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:20:03.169672 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.169652 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2a6efd-ce05-4b81-96c7-637db360b341-kube-api-access-t2qh6" (OuterVolumeSpecName: "kube-api-access-t2qh6") pod "eb2a6efd-ce05-4b81-96c7-637db360b341" (UID: "eb2a6efd-ce05-4b81-96c7-637db360b341"). InnerVolumeSpecName "kube-api-access-t2qh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:20:03.268090 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.268062 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qh6\" (UniqueName: \"kubernetes.io/projected/eb2a6efd-ce05-4b81-96c7-637db360b341-kube-api-access-t2qh6\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:03.268090 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.268088 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2a6efd-ce05-4b81-96c7-637db360b341-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:03.733535 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.733500 2582 generic.go:356] "Generic (PLEG): container finished" podID="ecd1f3f6-550f-4bac-960b-a61c0656ba1e" containerID="f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1" exitCode=0 Jul 29 21:20:03.733830 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.733554 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" Jul 29 21:20:03.733830 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.733588 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" event={"ID":"ecd1f3f6-550f-4bac-960b-a61c0656ba1e","Type":"ContainerDied","Data":"f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1"} Jul 29 21:20:03.733830 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.733652 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg" event={"ID":"ecd1f3f6-550f-4bac-960b-a61c0656ba1e","Type":"ContainerDied","Data":"373c30ac12eef09ccca832509a562b6f0670165e9a6a0af3adeac19a763ba039"} Jul 29 21:20:03.733830 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.733675 2582 scope.go:117] "RemoveContainer" containerID="f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1" Jul 29 21:20:03.735254 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.735222 2582 generic.go:356] "Generic (PLEG): container finished" podID="eb2a6efd-ce05-4b81-96c7-637db360b341" containerID="c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa" exitCode=0 Jul 29 21:20:03.735372 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.735256 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" event={"ID":"eb2a6efd-ce05-4b81-96c7-637db360b341","Type":"ContainerDied","Data":"c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa"} Jul 29 21:20:03.735372 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.735291 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" Jul 29 21:20:03.735372 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.735310 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6" event={"ID":"eb2a6efd-ce05-4b81-96c7-637db360b341","Type":"ContainerDied","Data":"cfc7d525a98a91e97eee09e579e4386236920118d5710632f088cf50c99f2c20"} Jul 29 21:20:03.743288 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.743264 2582 scope.go:117] "RemoveContainer" containerID="f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1" Jul 29 21:20:03.743606 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:20:03.743581 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1\": container with ID starting with f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1 not found: ID does not exist" containerID="f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1" Jul 29 21:20:03.743702 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.743621 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1"} err="failed to get container status \"f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1\": rpc error: code = NotFound desc = could not find container \"f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1\": container with ID starting with f50e5d6a10a42993337a607e85cb75e937966219de526890a439281e279756d1 not found: ID does not exist" Jul 29 21:20:03.743702 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.743645 2582 scope.go:117] "RemoveContainer" containerID="c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa" Jul 29 21:20:03.752726 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.752707 2582 scope.go:117] "RemoveContainer" containerID="c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa" Jul 29 21:20:03.753052 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:20:03.753004 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa\": container with ID starting with c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa not found: ID does not exist" containerID="c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa" Jul 29 21:20:03.753149 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.753062 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa"} err="failed to get container status \"c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa\": rpc error: code = NotFound desc = could not find container \"c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa\": container with ID starting with c755f2135abbc39bc611d55020908a131b0418ec02c72fbb444f6799b55ed2aa not found: ID does not exist" Jul 29 21:20:03.761281 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.761259 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg"] Jul 29 21:20:03.767151 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.767128 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e74c91e4-5ccd9b8d88-vmlpg"] Jul 29 21:20:03.779747 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.779725 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6"] Jul 29 21:20:03.784816 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:03.784794 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e74c91e4-99fc99876-hq8c6"] Jul 29 21:20:04.163386 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:04.163342 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f"] Jul 29 21:20:04.163605 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:04.163582 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" containerID="cri-o://858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e" gracePeriod=30 Jul 29 21:20:04.493123 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:04.493093 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa" path="/var/lib/kubelet/pods/3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa/volumes" Jul 29 21:20:04.493494 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:04.493480 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb2a6efd-ce05-4b81-96c7-637db360b341" path="/var/lib/kubelet/pods/eb2a6efd-ce05-4b81-96c7-637db360b341/volumes" Jul 29 21:20:04.493784 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:04.493772 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecd1f3f6-550f-4bac-960b-a61c0656ba1e" path="/var/lib/kubelet/pods/ecd1f3f6-550f-4bac-960b-a61c0656ba1e/volumes" Jul 29 21:20:05.009189 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.009148 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv"] Jul 29 21:20:05.009707 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.009683 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa" containerName="revoke-keys" Jul 29 21:20:05.009877 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.009858 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa" containerName="revoke-keys" Jul 29 21:20:05.009959 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.009885 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" Jul 29 21:20:05.009959 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.009895 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" Jul 29 21:20:05.009959 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.009932 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb2a6efd-ce05-4b81-96c7-637db360b341" containerName="payload-pre-processing" Jul 29 21:20:05.009959 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.009941 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb2a6efd-ce05-4b81-96c7-637db360b341" containerName="payload-pre-processing" Jul 29 21:20:05.009959 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.009954 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecd1f3f6-550f-4bac-960b-a61c0656ba1e" containerName="payload-processing" Jul 29 21:20:05.010158 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.009963 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecd1f3f6-550f-4bac-960b-a61c0656ba1e" containerName="payload-processing" Jul 29 21:20:05.010158 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.010079 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="753af74e-2e36-4324-aeca-e627b40eaee5" containerName="istio-proxy" Jul 29 21:20:05.010158 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.010093 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed0548b-ebe5-4ee5-bbcd-3edd6aaceeaa" containerName="revoke-keys" Jul 29 21:20:05.010158 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.010103 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecd1f3f6-550f-4bac-960b-a61c0656ba1e" containerName="payload-processing" Jul 29 21:20:05.010158 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.010115 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb2a6efd-ce05-4b81-96c7-637db360b341" containerName="payload-pre-processing" Jul 29 21:20:05.014351 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.014324 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.017975 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.017954 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f1022d1c-openshift-default-dockercfg-8fp5w\"" Jul 29 21:20:05.030445 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.030415 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv"] Jul 29 21:20:05.086834 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.086799 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-istio-data\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.086834 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.086838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-istio-envoy\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.087243 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.086873 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-credential-socket\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.087243 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.086990 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc495e7-12f4-4338-b58f-e718e1237049-istiod-ca-cert\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.087243 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.087115 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfbg5\" (UniqueName: \"kubernetes.io/projected/cbc495e7-12f4-4338-b58f-e718e1237049-kube-api-access-lfbg5\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.087243 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.087166 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc495e7-12f4-4338-b58f-e718e1237049-istio-podinfo\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.087243 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.087208 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-workload-socket\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.087432 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.087268 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc495e7-12f4-4338-b58f-e718e1237049-istio-token\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.087432 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.087290 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-workload-certs\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.188092 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbg5\" (UniqueName: \"kubernetes.io/projected/cbc495e7-12f4-4338-b58f-e718e1237049-kube-api-access-lfbg5\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.188284 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188117 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc495e7-12f4-4338-b58f-e718e1237049-istio-podinfo\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.188284 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188140 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-workload-socket\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.188284 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188167 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc495e7-12f4-4338-b58f-e718e1237049-istio-token\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.188473 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188287 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-workload-certs\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.188473 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188343 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-istio-data\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.188473 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188373 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-istio-envoy\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.188473 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188428 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-credential-socket\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.188696 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188488 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc495e7-12f4-4338-b58f-e718e1237049-istiod-ca-cert\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.188696 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188553 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-workload-socket\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.188793 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188700 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-workload-certs\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.188793 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188726 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-istio-data\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.189168 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.188944 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-credential-socket\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.189252 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.189174 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc495e7-12f4-4338-b58f-e718e1237049-istiod-ca-cert\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.190818 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.190792 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-istio-envoy\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.190920 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.190892 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc495e7-12f4-4338-b58f-e718e1237049-istio-podinfo\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.206837 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.206802 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc495e7-12f4-4338-b58f-e718e1237049-istio-token\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.206837 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.206820 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbg5\" (UniqueName: \"kubernetes.io/projected/cbc495e7-12f4-4338-b58f-e718e1237049-kube-api-access-lfbg5\") pod \"e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.327161 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.327067 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:05.471921 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.471897 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv"] Jul 29 21:20:05.473946 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:20:05.473923 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc495e7_12f4_4338_b58f_e718e1237049.slice/crio-c1907d5d39c74d86499572cb717cfdee0d022bfab531ff71ba1d3af68377b8f3 WatchSource:0}: Error finding container c1907d5d39c74d86499572cb717cfdee0d022bfab531ff71ba1d3af68377b8f3: Status 404 returned error can't find the container with id c1907d5d39c74d86499572cb717cfdee0d022bfab531ff71ba1d3af68377b8f3 Jul 29 21:20:05.476131 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.476099 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:20:05.476250 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.476163 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:20:05.476250 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.476194 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:20:05.750753 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.750712 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" event={"ID":"cbc495e7-12f4-4338-b58f-e718e1237049","Type":"ContainerStarted","Data":"dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0"} Jul 29 21:20:05.750941 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.750781 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" event={"ID":"cbc495e7-12f4-4338-b58f-e718e1237049","Type":"ContainerStarted","Data":"c1907d5d39c74d86499572cb717cfdee0d022bfab531ff71ba1d3af68377b8f3"} Jul 29 21:20:05.774120 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:05.774060 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" podStartSLOduration=1.7740429359999998 podStartE2EDuration="1.774042936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:20:05.771185049 +0000 UTC m=+1953.894774890" watchObservedRunningTime="2026-07-29 21:20:05.774042936 +0000 UTC m=+1953.897632770" Jul 29 21:20:06.327604 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:06.327562 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:06.333046 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:06.332993 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:06.756633 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:06.756600 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:06.757710 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:06.757686 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:07.043852 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:07.043765 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:20:08.951816 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:08.951780 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5"] Jul 29 21:20:08.954789 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:08.954773 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" Jul 29 21:20:08.957498 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:08.957476 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f1022d1c-dockercfg-2b6jf\"" Jul 29 21:20:08.957663 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:08.957644 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f1022d1c\"" Jul 29 21:20:08.973409 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:08.973382 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5"] Jul 29 21:20:09.025760 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.025728 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5\" (UID: \"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" Jul 29 21:20:09.025974 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.025797 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hchx\" (UniqueName: \"kubernetes.io/projected/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3-kube-api-access-2hchx\") pod \"payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5\" (UID: \"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" Jul 29 21:20:09.127273 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.127235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hchx\" (UniqueName: \"kubernetes.io/projected/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3-kube-api-access-2hchx\") pod \"payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5\" (UID: \"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" Jul 29 21:20:09.127520 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.127310 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5\" (UID: \"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" Jul 29 21:20:09.127995 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.127976 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5\" (UID: \"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" Jul 29 21:20:09.143199 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.143174 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hchx\" (UniqueName: \"kubernetes.io/projected/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3-kube-api-access-2hchx\") pod \"payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5\" (UID: \"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" Jul 29 21:20:09.265597 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.265562 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" Jul 29 21:20:09.432019 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.431992 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5"] Jul 29 21:20:09.433559 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:20:09.433533 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce844a0_28ad_4b2e_a5ab_e5c7f60a8fc3.slice/crio-d694a1b64c10bda9c8062851c06ee01eb6fd20b5a267e713fe8e728ec0ed14a0 WatchSource:0}: Error finding container d694a1b64c10bda9c8062851c06ee01eb6fd20b5a267e713fe8e728ec0ed14a0: Status 404 returned error can't find the container with id d694a1b64c10bda9c8062851c06ee01eb6fd20b5a267e713fe8e728ec0ed14a0 Jul 29 21:20:09.436103 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.436072 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:20:09.531285 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531190 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-token\") pod \"778953bd-eed8-4ad5-ac7d-d38f0e504830\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " Jul 29 21:20:09.531285 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531232 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-workload-socket\") pod \"778953bd-eed8-4ad5-ac7d-d38f0e504830\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " Jul 29 21:20:09.531285 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531257 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-envoy\") pod \"778953bd-eed8-4ad5-ac7d-d38f0e504830\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " Jul 29 21:20:09.531608 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531291 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-workload-certs\") pod \"778953bd-eed8-4ad5-ac7d-d38f0e504830\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " Jul 29 21:20:09.531608 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531345 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778953bd-eed8-4ad5-ac7d-d38f0e504830-istiod-ca-cert\") pod \"778953bd-eed8-4ad5-ac7d-d38f0e504830\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " Jul 29 21:20:09.531608 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531419 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-credential-socket\") pod \"778953bd-eed8-4ad5-ac7d-d38f0e504830\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " Jul 29 21:20:09.531608 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531437 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-data\") pod \"778953bd-eed8-4ad5-ac7d-d38f0e504830\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " Jul 29 21:20:09.531608 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531471 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz8f9\" (UniqueName: \"kubernetes.io/projected/778953bd-eed8-4ad5-ac7d-d38f0e504830-kube-api-access-vz8f9\") pod \"778953bd-eed8-4ad5-ac7d-d38f0e504830\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " Jul 29 21:20:09.531608 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531495 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-podinfo\") pod \"778953bd-eed8-4ad5-ac7d-d38f0e504830\" (UID: \"778953bd-eed8-4ad5-ac7d-d38f0e504830\") " Jul 29 21:20:09.531930 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531464 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "778953bd-eed8-4ad5-ac7d-d38f0e504830" (UID: "778953bd-eed8-4ad5-ac7d-d38f0e504830"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:20:09.531930 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531618 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "778953bd-eed8-4ad5-ac7d-d38f0e504830" (UID: "778953bd-eed8-4ad5-ac7d-d38f0e504830"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:20:09.531930 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531628 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "778953bd-eed8-4ad5-ac7d-d38f0e504830" (UID: "778953bd-eed8-4ad5-ac7d-d38f0e504830"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:20:09.531930 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.531821 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:09.532358 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.532085 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778953bd-eed8-4ad5-ac7d-d38f0e504830-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "778953bd-eed8-4ad5-ac7d-d38f0e504830" (UID: "778953bd-eed8-4ad5-ac7d-d38f0e504830"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:20:09.532422 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.532372 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-data" (OuterVolumeSpecName: "istio-data") pod "778953bd-eed8-4ad5-ac7d-d38f0e504830" (UID: "778953bd-eed8-4ad5-ac7d-d38f0e504830"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:20:09.533580 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.533558 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "778953bd-eed8-4ad5-ac7d-d38f0e504830" (UID: "778953bd-eed8-4ad5-ac7d-d38f0e504830"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:20:09.533706 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.533640 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-token" (OuterVolumeSpecName: "istio-token") pod "778953bd-eed8-4ad5-ac7d-d38f0e504830" (UID: "778953bd-eed8-4ad5-ac7d-d38f0e504830"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:20:09.534229 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.534206 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "778953bd-eed8-4ad5-ac7d-d38f0e504830" (UID: "778953bd-eed8-4ad5-ac7d-d38f0e504830"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:20:09.534229 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.534214 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778953bd-eed8-4ad5-ac7d-d38f0e504830-kube-api-access-vz8f9" (OuterVolumeSpecName: "kube-api-access-vz8f9") pod "778953bd-eed8-4ad5-ac7d-d38f0e504830" (UID: "778953bd-eed8-4ad5-ac7d-d38f0e504830"). InnerVolumeSpecName "kube-api-access-vz8f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:20:09.633229 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.633189 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz8f9\" (UniqueName: \"kubernetes.io/projected/778953bd-eed8-4ad5-ac7d-d38f0e504830-kube-api-access-vz8f9\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:09.633229 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.633223 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:09.633229 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.633232 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:09.633229 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.633240 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:09.633503 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.633248 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:09.633503 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.633256 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778953bd-eed8-4ad5-ac7d-d38f0e504830-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:09.633503 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.633264 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:09.633503 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.633271 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778953bd-eed8-4ad5-ac7d-d38f0e504830-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:09.779482 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.779445 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" event={"ID":"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3","Type":"ContainerStarted","Data":"0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233"} Jul 29 21:20:09.779709 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.779520 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" Jul 29 21:20:09.779709 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.779546 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" event={"ID":"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3","Type":"ContainerStarted","Data":"d694a1b64c10bda9c8062851c06ee01eb6fd20b5a267e713fe8e728ec0ed14a0"} Jul 29 21:20:09.781259 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.781232 2582 generic.go:356] "Generic (PLEG): container finished" podID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerID="858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e" exitCode=0 Jul 29 21:20:09.781402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.781276 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" Jul 29 21:20:09.781402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.781288 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" event={"ID":"778953bd-eed8-4ad5-ac7d-d38f0e504830","Type":"ContainerDied","Data":"858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e"} Jul 29 21:20:09.781402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.781335 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f" event={"ID":"778953bd-eed8-4ad5-ac7d-d38f0e504830","Type":"ContainerDied","Data":"6b596ad6053bb2ece75526a53334f3fe7344eea5751972735a3c1e1d21b23b9e"} Jul 29 21:20:09.781402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.781366 2582 scope.go:117] "RemoveContainer" containerID="858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e" Jul 29 21:20:09.790785 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.790765 2582 scope.go:117] "RemoveContainer" containerID="4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93" Jul 29 21:20:09.797480 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.797429 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" podStartSLOduration=1.797412336 podStartE2EDuration="1.797412336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:20:09.795917367 +0000 UTC m=+1957.919507209" watchObservedRunningTime="2026-07-29 21:20:09.797412336 +0000 UTC m=+1957.921002216" Jul 29 21:20:09.801781 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.801664 2582 scope.go:117] "RemoveContainer" containerID="858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e" Jul 29 21:20:09.802094 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:20:09.802068 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e\": container with ID starting with 858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e not found: ID does not exist" containerID="858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e" Jul 29 21:20:09.802231 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.802101 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e"} err="failed to get container status \"858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e\": rpc error: code = NotFound desc = could not find container \"858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e\": container with ID starting with 858feedde734537bfaeba9f4c0ca6d7e9b80c7cbc835765747e4f8dd5e5a865e not found: ID does not exist" Jul 29 21:20:09.802231 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.802127 2582 scope.go:117] "RemoveContainer" containerID="4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93" Jul 29 21:20:09.802438 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:20:09.802420 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93\": container with ID starting with 4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93 not found: ID does not exist" containerID="4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93" Jul 29 21:20:09.802526 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.802444 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93"} err="failed to get container status \"4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93\": rpc error: code = NotFound desc = could not find container \"4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93\": container with ID starting with 4fb507661e556e529f508adc8c69488561f96c99e254560e8cb2ad82dfba2a93 not found: ID does not exist" Jul 29 21:20:09.812197 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.812171 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f"] Jul 29 21:20:09.817993 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:09.817967 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e74c91e4-openshift-default-79d46ff95b-cgr5f"] Jul 29 21:20:10.494044 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:10.494004 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" path="/var/lib/kubelet/pods/778953bd-eed8-4ad5-ac7d-d38f0e504830/volumes" Jul 29 21:20:19.051762 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:19.051722 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:20:20.790497 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:20.790458 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" Jul 29 21:20:21.758577 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:21.758541 2582 patch_prober.go:28] interesting pod/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 29 21:20:21.758789 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:21.758610 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 29 21:20:21.857932 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:21.857900 2582 generic.go:356] "Generic (PLEG): container finished" podID="cbc495e7-12f4-4338-b58f-e718e1237049" containerID="dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0" exitCode=0 Jul 29 21:20:21.858347 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:21.857968 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" event={"ID":"cbc495e7-12f4-4338-b58f-e718e1237049","Type":"ContainerDied","Data":"dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0"} Jul 29 21:20:21.858415 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:21.858398 2582 scope.go:117] "RemoveContainer" containerID="dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0" Jul 29 21:20:21.859399 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:21.859372 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:20:21.859453 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:21.859440 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:20:21.859489 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:21.859469 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:20:22.865156 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:22.865122 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" event={"ID":"cbc495e7-12f4-4338-b58f-e718e1237049","Type":"ContainerStarted","Data":"a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814"} Jul 29 21:20:23.327519 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:23.327479 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:23.327766 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:23.327740 2582 patch_prober.go:28] interesting pod/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 29 21:20:23.327825 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:23.327806 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 29 21:20:24.327826 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:24.327783 2582 patch_prober.go:28] interesting pod/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 29 21:20:24.328293 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:24.327857 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 29 21:20:25.328380 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:25.328334 2582 patch_prober.go:28] interesting pod/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 29 21:20:25.328805 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:25.328413 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 29 21:20:26.327670 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:26.327629 2582 patch_prober.go:28] interesting pod/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 29 21:20:26.327870 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:26.327692 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 29 21:20:27.332722 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:27.332692 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:27.333182 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:27.333062 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:27.333806 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:27.333786 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:32.064917 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.064876 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2"] Jul 29 21:20:32.065405 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.065387 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" Jul 29 21:20:32.065452 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.065407 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" Jul 29 21:20:32.065452 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.065425 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" Jul 29 21:20:32.065452 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.065430 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" Jul 29 21:20:32.065553 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.065504 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" Jul 29 21:20:32.065553 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.065511 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="778953bd-eed8-4ad5-ac7d-d38f0e504830" containerName="istio-proxy" Jul 29 21:20:32.068814 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.068797 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2" Jul 29 21:20:32.072832 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.072807 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:20:32.073003 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.072808 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\"" Jul 29 21:20:32.077911 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.077888 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2"] Jul 29 21:20:32.158557 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.158513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd3035-9662-4890-a8a8-6d8f2bedecc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2\" (UID: \"a9cd3035-9662-4890-a8a8-6d8f2bedecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2" Jul 29 21:20:32.259633 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.259600 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd3035-9662-4890-a8a8-6d8f2bedecc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2\" (UID: \"a9cd3035-9662-4890-a8a8-6d8f2bedecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2" Jul 29 21:20:32.260309 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.260288 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd3035-9662-4890-a8a8-6d8f2bedecc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2\" (UID: \"a9cd3035-9662-4890-a8a8-6d8f2bedecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2" Jul 29 21:20:32.384107 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.383997 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\"" Jul 29 21:20:32.392223 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.392194 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2" Jul 29 21:20:32.534950 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.534766 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2"] Jul 29 21:20:32.537295 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:20:32.537261 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cd3035_9662_4890_a8a8_6d8f2bedecc8.slice/crio-d85f82808a3819bf5958dbc080b0d5619f55cf74f122aedf267b565ca38df6f0 WatchSource:0}: Error finding container d85f82808a3819bf5958dbc080b0d5619f55cf74f122aedf267b565ca38df6f0: Status 404 returned error can't find the container with id d85f82808a3819bf5958dbc080b0d5619f55cf74f122aedf267b565ca38df6f0 Jul 29 21:20:32.919870 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.919826 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2" event={"ID":"a9cd3035-9662-4890-a8a8-6d8f2bedecc8","Type":"ContainerStarted","Data":"120c23b09a47c1818dcbae939246cd0f720c3bc8f9737448975a074de17d9fd0"} Jul 29 21:20:32.920074 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.919894 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2" event={"ID":"a9cd3035-9662-4890-a8a8-6d8f2bedecc8","Type":"ContainerStarted","Data":"d85f82808a3819bf5958dbc080b0d5619f55cf74f122aedf267b565ca38df6f0"} Jul 29 21:20:32.935172 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:32.935121 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2" podStartSLOduration=0.935105445 podStartE2EDuration="935.105445ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:20:32.934425222 +0000 UTC m=+1981.058015067" watchObservedRunningTime="2026-07-29 21:20:32.935105445 +0000 UTC m=+1981.058695287" Jul 29 21:20:33.927622 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:33.927585 2582 generic.go:356] "Generic (PLEG): container finished" podID="a9cd3035-9662-4890-a8a8-6d8f2bedecc8" containerID="120c23b09a47c1818dcbae939246cd0f720c3bc8f9737448975a074de17d9fd0" exitCode=0 Jul 29 21:20:33.928017 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:33.927676 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2" event={"ID":"a9cd3035-9662-4890-a8a8-6d8f2bedecc8","Type":"ContainerDied","Data":"120c23b09a47c1818dcbae939246cd0f720c3bc8f9737448975a074de17d9fd0"} Jul 29 21:20:35.080939 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:35.080910 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2" Jul 29 21:20:35.184723 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:35.184631 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd3035-9662-4890-a8a8-6d8f2bedecc8-maas-api-service-ca\") pod \"a9cd3035-9662-4890-a8a8-6d8f2bedecc8\" (UID: \"a9cd3035-9662-4890-a8a8-6d8f2bedecc8\") " Jul 29 21:20:35.185088 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:35.185058 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cd3035-9662-4890-a8a8-6d8f2bedecc8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9cd3035-9662-4890-a8a8-6d8f2bedecc8" (UID: "a9cd3035-9662-4890-a8a8-6d8f2bedecc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:20:35.285998 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:35.285961 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd3035-9662-4890-a8a8-6d8f2bedecc8-maas-api-service-ca\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:35.940985 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:35.940944 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2" Jul 29 21:20:35.940985 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:35.940959 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2" event={"ID":"a9cd3035-9662-4890-a8a8-6d8f2bedecc8","Type":"ContainerDied","Data":"d85f82808a3819bf5958dbc080b0d5619f55cf74f122aedf267b565ca38df6f0"} Jul 29 21:20:35.941241 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:35.941053 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d85f82808a3819bf5958dbc080b0d5619f55cf74f122aedf267b565ca38df6f0" Jul 29 21:20:38.912440 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:38.912405 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5"] Jul 29 21:20:38.912912 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:38.912660 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" podUID="5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3" containerName="payload-pre-processing" containerID="cri-o://0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233" gracePeriod=30 Jul 29 21:20:39.057693 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.057648 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2"] Jul 29 21:20:39.059930 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.059904 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f1022d1c-hmpv2"] Jul 29 21:20:39.160312 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.160289 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" Jul 29 21:20:39.194680 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.194593 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv"] Jul 29 21:20:39.195201 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.195148 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" containerID="cri-o://a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814" gracePeriod=30 Jul 29 21:20:39.221325 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.221268 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3-plugins-config-volume\") pod \"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3\" (UID: \"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3\") " Jul 29 21:20:39.221510 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.221335 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hchx\" (UniqueName: \"kubernetes.io/projected/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3-kube-api-access-2hchx\") pod \"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3\" (UID: \"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3\") " Jul 29 21:20:39.222107 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.222078 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3" (UID: "5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:20:39.224197 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.224167 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3-kube-api-access-2hchx" (OuterVolumeSpecName: "kube-api-access-2hchx") pod "5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3" (UID: "5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3"). InnerVolumeSpecName "kube-api-access-2hchx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:20:39.322796 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.322761 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:39.322796 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.322793 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hchx\" (UniqueName: \"kubernetes.io/projected/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3-kube-api-access-2hchx\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:39.965999 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.965963 2582 generic.go:356] "Generic (PLEG): container finished" podID="5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3" containerID="0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233" exitCode=0 Jul 29 21:20:39.966453 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.966049 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" Jul 29 21:20:39.966453 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.966051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" event={"ID":"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3","Type":"ContainerDied","Data":"0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233"} Jul 29 21:20:39.966453 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.966106 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5" event={"ID":"5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3","Type":"ContainerDied","Data":"d694a1b64c10bda9c8062851c06ee01eb6fd20b5a267e713fe8e728ec0ed14a0"} Jul 29 21:20:39.966453 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.966136 2582 scope.go:117] "RemoveContainer" containerID="0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233" Jul 29 21:20:39.978600 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.978578 2582 scope.go:117] "RemoveContainer" containerID="0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233" Jul 29 21:20:39.978969 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:20:39.978947 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233\": container with ID starting with 0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233 not found: ID does not exist" containerID="0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233" Jul 29 21:20:39.979076 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.978982 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233"} err="failed to get container status \"0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233\": rpc error: code = NotFound desc = could not find container \"0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233\": container with ID starting with 0f9f4bc94f1376d0ec141f8fa01cbd9c6edfd3c16857c10c94f75123065d3233 not found: ID does not exist" Jul 29 21:20:39.994244 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.994216 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5"] Jul 29 21:20:39.998506 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:39.998483 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f1022d1c-d86f47d96-55mz5"] Jul 29 21:20:40.493685 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:40.493650 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3" path="/var/lib/kubelet/pods/5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3/volumes" Jul 29 21:20:40.494119 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:40.494096 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9cd3035-9662-4890-a8a8-6d8f2bedecc8" path="/var/lib/kubelet/pods/a9cd3035-9662-4890-a8a8-6d8f2bedecc8/volumes" Jul 29 21:20:42.335827 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:42.335782 2582 patch_prober.go:28] interesting pod/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 29 21:20:42.336226 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:42.335862 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 21:20:42.659623 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:42.659539 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:20:44.491049 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.485851 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:44.575066 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575007 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-istio-envoy\") pod \"cbc495e7-12f4-4338-b58f-e718e1237049\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " Jul 29 21:20:44.575234 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575107 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-workload-socket\") pod \"cbc495e7-12f4-4338-b58f-e718e1237049\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " Jul 29 21:20:44.575234 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575189 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc495e7-12f4-4338-b58f-e718e1237049-istio-podinfo\") pod \"cbc495e7-12f4-4338-b58f-e718e1237049\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " Jul 29 21:20:44.575307 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575240 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-istio-data\") pod \"cbc495e7-12f4-4338-b58f-e718e1237049\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " Jul 29 21:20:44.575307 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575266 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfbg5\" (UniqueName: \"kubernetes.io/projected/cbc495e7-12f4-4338-b58f-e718e1237049-kube-api-access-lfbg5\") pod \"cbc495e7-12f4-4338-b58f-e718e1237049\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " Jul 29 21:20:44.575307 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575290 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc495e7-12f4-4338-b58f-e718e1237049-istio-token\") pod \"cbc495e7-12f4-4338-b58f-e718e1237049\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " Jul 29 21:20:44.575432 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575317 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-credential-socket\") pod \"cbc495e7-12f4-4338-b58f-e718e1237049\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " Jul 29 21:20:44.575432 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575341 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc495e7-12f4-4338-b58f-e718e1237049-istiod-ca-cert\") pod \"cbc495e7-12f4-4338-b58f-e718e1237049\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " Jul 29 21:20:44.575432 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575374 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-workload-certs\") pod \"cbc495e7-12f4-4338-b58f-e718e1237049\" (UID: \"cbc495e7-12f4-4338-b58f-e718e1237049\") " Jul 29 21:20:44.575432 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575399 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cbc495e7-12f4-4338-b58f-e718e1237049" (UID: "cbc495e7-12f4-4338-b58f-e718e1237049"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:20:44.575695 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575672 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:44.575795 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575752 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc495e7-12f4-4338-b58f-e718e1237049-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cbc495e7-12f4-4338-b58f-e718e1237049" (UID: "cbc495e7-12f4-4338-b58f-e718e1237049"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:20:44.575902 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.575874 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cbc495e7-12f4-4338-b58f-e718e1237049" (UID: "cbc495e7-12f4-4338-b58f-e718e1237049"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:20:44.576116 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.576077 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cbc495e7-12f4-4338-b58f-e718e1237049" (UID: "cbc495e7-12f4-4338-b58f-e718e1237049"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:20:44.576116 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.576087 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-istio-data" (OuterVolumeSpecName: "istio-data") pod "cbc495e7-12f4-4338-b58f-e718e1237049" (UID: "cbc495e7-12f4-4338-b58f-e718e1237049"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:20:44.577587 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.577559 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cbc495e7-12f4-4338-b58f-e718e1237049" (UID: "cbc495e7-12f4-4338-b58f-e718e1237049"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:20:44.577681 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.577655 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc495e7-12f4-4338-b58f-e718e1237049-kube-api-access-lfbg5" (OuterVolumeSpecName: "kube-api-access-lfbg5") pod "cbc495e7-12f4-4338-b58f-e718e1237049" (UID: "cbc495e7-12f4-4338-b58f-e718e1237049"). InnerVolumeSpecName "kube-api-access-lfbg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:20:44.577749 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.577727 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc495e7-12f4-4338-b58f-e718e1237049-istio-token" (OuterVolumeSpecName: "istio-token") pod "cbc495e7-12f4-4338-b58f-e718e1237049" (UID: "cbc495e7-12f4-4338-b58f-e718e1237049"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:20:44.577794 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.577742 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbc495e7-12f4-4338-b58f-e718e1237049-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cbc495e7-12f4-4338-b58f-e718e1237049" (UID: "cbc495e7-12f4-4338-b58f-e718e1237049"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:20:44.677154 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.677118 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc495e7-12f4-4338-b58f-e718e1237049-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:44.677154 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.677148 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:44.677154 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.677158 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfbg5\" (UniqueName: \"kubernetes.io/projected/cbc495e7-12f4-4338-b58f-e718e1237049-kube-api-access-lfbg5\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:44.677154 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.677167 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc495e7-12f4-4338-b58f-e718e1237049-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:44.677432 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.677176 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:44.677432 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.677184 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc495e7-12f4-4338-b58f-e718e1237049-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:44.677432 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.677193 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:44.677432 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.677202 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc495e7-12f4-4338-b58f-e718e1237049-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:20:44.998901 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.998859 2582 generic.go:356] "Generic (PLEG): container finished" podID="cbc495e7-12f4-4338-b58f-e718e1237049" containerID="a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814" exitCode=0 Jul 29 21:20:44.999103 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.998961 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" event={"ID":"cbc495e7-12f4-4338-b58f-e718e1237049","Type":"ContainerDied","Data":"a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814"} Jul 29 21:20:44.999103 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.998986 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" Jul 29 21:20:44.999103 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.999040 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv" event={"ID":"cbc495e7-12f4-4338-b58f-e718e1237049","Type":"ContainerDied","Data":"c1907d5d39c74d86499572cb717cfdee0d022bfab531ff71ba1d3af68377b8f3"} Jul 29 21:20:44.999103 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:44.999053 2582 scope.go:117] "RemoveContainer" containerID="a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814" Jul 29 21:20:45.009217 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:45.009198 2582 scope.go:117] "RemoveContainer" containerID="dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0" Jul 29 21:20:45.018135 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:45.018113 2582 scope.go:117] "RemoveContainer" containerID="a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814" Jul 29 21:20:45.018416 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:20:45.018394 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814\": container with ID starting with a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814 not found: ID does not exist" containerID="a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814" Jul 29 21:20:45.018478 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:45.018426 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814"} err="failed to get container status \"a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814\": rpc error: code = NotFound desc = could not find container \"a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814\": container with ID starting with a7953ed0dcab72061b29a45adde9ca2ca6e5594cc66589dcfa29448952305814 not found: ID does not exist" Jul 29 21:20:45.018478 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:45.018445 2582 scope.go:117] "RemoveContainer" containerID="dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0" Jul 29 21:20:45.018746 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:20:45.018728 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0\": container with ID starting with dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0 not found: ID does not exist" containerID="dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0" Jul 29 21:20:45.018801 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:45.018754 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0"} err="failed to get container status \"dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0\": rpc error: code = NotFound desc = could not find container \"dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0\": container with ID starting with dcdd4575cef66e9cb2576d3d6a6f5f72eaad4e4be5366799e35d103eb3f47aa0 not found: ID does not exist" Jul 29 21:20:45.021903 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:45.021880 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv"] Jul 29 21:20:45.028564 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:45.028542 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f1022d1c-openshift-default-8b8669bf6-9dlhv"] Jul 29 21:20:46.494150 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:20:46.494110 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" path="/var/lib/kubelet/pods/cbc495e7-12f4-4338-b58f-e718e1237049/volumes" Jul 29 21:22:05.452335 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:05.452234 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:22:16.457329 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:16.457286 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:22:24.009708 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.009673 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58"] Jul 29 21:22:24.010247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.010008 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3" containerName="payload-pre-processing" Jul 29 21:22:24.010247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.010038 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3" containerName="payload-pre-processing" Jul 29 21:22:24.010247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.010059 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" Jul 29 21:22:24.010247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.010065 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" Jul 29 21:22:24.010247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.010082 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cd3035-9662-4890-a8a8-6d8f2bedecc8" containerName="revoke-keys" Jul 29 21:22:24.010247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.010087 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cd3035-9662-4890-a8a8-6d8f2bedecc8" containerName="revoke-keys" Jul 29 21:22:24.010247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.010143 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" Jul 29 21:22:24.010247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.010152 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ce844a0-28ad-4b2e-a5ab-e5c7f60a8fc3" containerName="payload-pre-processing" Jul 29 21:22:24.010247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.010158 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9cd3035-9662-4890-a8a8-6d8f2bedecc8" containerName="revoke-keys" Jul 29 21:22:24.012588 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.012560 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.019325 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.019297 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ccbf66ba-openshift-default-dockercfg-f9tbr\"" Jul 29 21:22:24.053286 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.053246 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58"] Jul 29 21:22:24.055634 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.055594 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-envoy\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.055805 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.055720 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-workload-socket\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.055805 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.055753 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-token\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.055805 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.055778 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istiod-ca-cert\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.055948 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.055889 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-data\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.056001 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.055947 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rcm8\" (UniqueName: \"kubernetes.io/projected/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-kube-api-access-9rcm8\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.056001 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.055989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-workload-certs\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.056143 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.056077 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-podinfo\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.056143 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.056117 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-credential-socket\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.156875 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.156835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-credential-socket\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.157134 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.156894 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-envoy\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.157134 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.156954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-workload-socket\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.157134 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.156972 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-token\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.157134 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.156994 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istiod-ca-cert\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.157134 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.157051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-data\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.157134 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.157095 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rcm8\" (UniqueName: \"kubernetes.io/projected/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-kube-api-access-9rcm8\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.157134 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.157123 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-workload-certs\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.157491 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.157286 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-podinfo\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.157491 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.157410 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-workload-socket\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.157491 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.157466 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-credential-socket\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.157657 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.157558 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-workload-certs\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.157942 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.157912 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istiod-ca-cert\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.158122 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.157963 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-data\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.159887 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.159862 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-envoy\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.159887 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.159877 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-podinfo\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.172267 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.172235 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-token\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.172526 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.172507 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rcm8\" (UniqueName: \"kubernetes.io/projected/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-kube-api-access-9rcm8\") pod \"e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.325437 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.325304 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:24.480572 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.480546 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58"] Jul 29 21:22:24.483320 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:22:24.483288 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f20ed4_21e1_4f67_ad2e_c641a3e6a3db.slice/crio-88acea9fda6edd182c78ad637866380a23885d98df9e96a176fb93db54b3f346 WatchSource:0}: Error finding container 88acea9fda6edd182c78ad637866380a23885d98df9e96a176fb93db54b3f346: Status 404 returned error can't find the container with id 88acea9fda6edd182c78ad637866380a23885d98df9e96a176fb93db54b3f346 Jul 29 21:22:24.485645 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.485607 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:22:24.485748 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.485681 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:22:24.485748 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.485710 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:22:24.535852 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:24.535821 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" event={"ID":"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db","Type":"ContainerStarted","Data":"88acea9fda6edd182c78ad637866380a23885d98df9e96a176fb93db54b3f346"} Jul 29 21:22:25.544088 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:25.544017 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" event={"ID":"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db","Type":"ContainerStarted","Data":"a59155714ea75c1bcdfefc4e1a8b6e8c8377bb5160170f4c18114de0cff83d62"} Jul 29 21:22:25.565016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:25.564965 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" podStartSLOduration=2.564948156 podStartE2EDuration="2.564948156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:22:25.563647785 +0000 UTC m=+2093.687237628" watchObservedRunningTime="2026-07-29 21:22:25.564948156 +0000 UTC m=+2093.688537976" Jul 29 21:22:26.326306 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:26.326264 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:26.331175 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:26.331151 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:26.549089 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:26.549060 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:26.550282 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:26.550264 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:27.770895 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:27.770859 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:22:29.384899 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.384865 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85"] Jul 29 21:22:29.385293 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.385269 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" Jul 29 21:22:29.385293 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.385282 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" Jul 29 21:22:29.385379 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.385339 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc495e7-12f4-4338-b58f-e718e1237049" containerName="istio-proxy" Jul 29 21:22:29.388127 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.388108 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" Jul 29 21:22:29.390790 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.390766 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ccbf66ba-dockercfg-jm2f6\"" Jul 29 21:22:29.391120 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.391089 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ccbf66ba\"" Jul 29 21:22:29.399504 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.399474 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85"] Jul 29 21:22:29.419046 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.418665 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp"] Jul 29 21:22:29.423303 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.423274 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" Jul 29 21:22:29.433657 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.433624 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp"] Jul 29 21:22:29.517863 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.517822 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190fb518-f280-40e1-8979-aed9540e794c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85\" (UID: \"190fb518-f280-40e1-8979-aed9540e794c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" Jul 29 21:22:29.518173 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.518149 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88sx\" (UniqueName: \"kubernetes.io/projected/02748163-23e7-4eb4-847c-709476f093e1-kube-api-access-d88sx\") pod \"payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp\" (UID: \"02748163-23e7-4eb4-847c-709476f093e1\") " pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" Jul 29 21:22:29.518378 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.518361 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02748163-23e7-4eb4-847c-709476f093e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp\" (UID: \"02748163-23e7-4eb4-847c-709476f093e1\") " pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" Jul 29 21:22:29.518501 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.518487 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8jsr\" (UniqueName: \"kubernetes.io/projected/190fb518-f280-40e1-8979-aed9540e794c-kube-api-access-h8jsr\") pod \"payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85\" (UID: \"190fb518-f280-40e1-8979-aed9540e794c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" Jul 29 21:22:29.619355 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.619312 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02748163-23e7-4eb4-847c-709476f093e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp\" (UID: \"02748163-23e7-4eb4-847c-709476f093e1\") " pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" Jul 29 21:22:29.619355 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.619359 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8jsr\" (UniqueName: \"kubernetes.io/projected/190fb518-f280-40e1-8979-aed9540e794c-kube-api-access-h8jsr\") pod \"payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85\" (UID: \"190fb518-f280-40e1-8979-aed9540e794c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" Jul 29 21:22:29.619626 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.619454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190fb518-f280-40e1-8979-aed9540e794c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85\" (UID: \"190fb518-f280-40e1-8979-aed9540e794c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" Jul 29 21:22:29.619626 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.619507 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88sx\" (UniqueName: \"kubernetes.io/projected/02748163-23e7-4eb4-847c-709476f093e1-kube-api-access-d88sx\") pod \"payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp\" (UID: \"02748163-23e7-4eb4-847c-709476f093e1\") " pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" Jul 29 21:22:29.620225 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.620201 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02748163-23e7-4eb4-847c-709476f093e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp\" (UID: \"02748163-23e7-4eb4-847c-709476f093e1\") " pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" Jul 29 21:22:29.620376 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.620351 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190fb518-f280-40e1-8979-aed9540e794c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85\" (UID: \"190fb518-f280-40e1-8979-aed9540e794c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" Jul 29 21:22:29.628674 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.628641 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88sx\" (UniqueName: \"kubernetes.io/projected/02748163-23e7-4eb4-847c-709476f093e1-kube-api-access-d88sx\") pod \"payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp\" (UID: \"02748163-23e7-4eb4-847c-709476f093e1\") " pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" Jul 29 21:22:29.628830 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.628641 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8jsr\" (UniqueName: \"kubernetes.io/projected/190fb518-f280-40e1-8979-aed9540e794c-kube-api-access-h8jsr\") pod \"payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85\" (UID: \"190fb518-f280-40e1-8979-aed9540e794c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" Jul 29 21:22:29.701635 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.701538 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" Jul 29 21:22:29.737263 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.736970 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" Jul 29 21:22:29.884438 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.884347 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85"] Jul 29 21:22:29.888277 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:22:29.888236 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190fb518_f280_40e1_8979_aed9540e794c.slice/crio-595f0a1f85e21d564f844d52430e9a2d971e196f6597ce8bfe869b0829a5a964 WatchSource:0}: Error finding container 595f0a1f85e21d564f844d52430e9a2d971e196f6597ce8bfe869b0829a5a964: Status 404 returned error can't find the container with id 595f0a1f85e21d564f844d52430e9a2d971e196f6597ce8bfe869b0829a5a964 Jul 29 21:22:29.931843 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:29.931810 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp"] Jul 29 21:22:29.934394 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:22:29.934364 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02748163_23e7_4eb4_847c_709476f093e1.slice/crio-fa1534f781d9c79202aef7f781cdc629cb55e25486d05b9fa6f4b077a92194f2 WatchSource:0}: Error finding container fa1534f781d9c79202aef7f781cdc629cb55e25486d05b9fa6f4b077a92194f2: Status 404 returned error can't find the container with id fa1534f781d9c79202aef7f781cdc629cb55e25486d05b9fa6f4b077a92194f2 Jul 29 21:22:30.574110 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:30.574072 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" event={"ID":"02748163-23e7-4eb4-847c-709476f093e1","Type":"ContainerStarted","Data":"e5e295f85c97dd94696d2406439e5f3fa8d72f84f29d578589091f881d9c4854"} Jul 29 21:22:30.574593 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:30.574131 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" event={"ID":"02748163-23e7-4eb4-847c-709476f093e1","Type":"ContainerStarted","Data":"fa1534f781d9c79202aef7f781cdc629cb55e25486d05b9fa6f4b077a92194f2"} Jul 29 21:22:30.574593 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:30.574159 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" Jul 29 21:22:30.575932 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:30.575905 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" event={"ID":"190fb518-f280-40e1-8979-aed9540e794c","Type":"ContainerStarted","Data":"9e13fa70f87da0427a27bb18e86a57186608af2f3803b7e4b46d0abcf72457aa"} Jul 29 21:22:30.576070 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:30.575956 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" event={"ID":"190fb518-f280-40e1-8979-aed9540e794c","Type":"ContainerStarted","Data":"595f0a1f85e21d564f844d52430e9a2d971e196f6597ce8bfe869b0829a5a964"} Jul 29 21:22:30.576070 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:30.576048 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" Jul 29 21:22:30.597954 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:30.597895 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" podStartSLOduration=1.597860539 podStartE2EDuration="1.597860539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:22:30.593680341 +0000 UTC m=+2098.717270186" watchObservedRunningTime="2026-07-29 21:22:30.597860539 +0000 UTC m=+2098.721450384" Jul 29 21:22:30.613325 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:30.613258 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" podStartSLOduration=1.6132425179999998 podStartE2EDuration="1.613242518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:22:30.610653203 +0000 UTC m=+2098.734243046" watchObservedRunningTime="2026-07-29 21:22:30.613242518 +0000 UTC m=+2098.736832361" Jul 29 21:22:41.584241 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:41.584207 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" Jul 29 21:22:41.584671 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:41.584289 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" Jul 29 21:22:41.857205 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:41.857114 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:22:42.287456 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:42.287053 2582 patch_prober.go:28] interesting pod/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": read tcp 10.134.0.2:44718->10.134.0.64:15021: read: connection reset by peer" start-of-body= Jul 29 21:22:42.287456 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:42.287139 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": read tcp 10.134.0.2:44718->10.134.0.64:15021: read: connection reset by peer" Jul 29 21:22:42.656420 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:42.656329 2582 generic.go:356] "Generic (PLEG): container finished" podID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerID="a59155714ea75c1bcdfefc4e1a8b6e8c8377bb5160170f4c18114de0cff83d62" exitCode=0 Jul 29 21:22:42.656420 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:42.656401 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" event={"ID":"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db","Type":"ContainerDied","Data":"a59155714ea75c1bcdfefc4e1a8b6e8c8377bb5160170f4c18114de0cff83d62"} Jul 29 21:22:42.656837 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:42.656821 2582 scope.go:117] "RemoveContainer" containerID="a59155714ea75c1bcdfefc4e1a8b6e8c8377bb5160170f4c18114de0cff83d62" Jul 29 21:22:42.657860 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:42.657833 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:22:42.657917 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:42.657898 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:22:42.657954 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:42.657928 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:22:43.663514 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:43.663475 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" event={"ID":"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db","Type":"ContainerStarted","Data":"e229cb4efbe148f6acc02cfec9434d2a98dd1e1ccb246d92089971f97b7e6f2f"} Jul 29 21:22:44.325646 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:44.325608 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:44.325876 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:44.325852 2582 patch_prober.go:28] interesting pod/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 29 21:22:44.325959 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:44.325910 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 29 21:22:45.326218 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:45.326180 2582 patch_prober.go:28] interesting pod/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 29 21:22:45.326641 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:45.326239 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 29 21:22:46.325822 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:46.325780 2582 patch_prober.go:28] interesting pod/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 29 21:22:46.326008 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:46.325853 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 29 21:22:47.326297 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:47.326256 2582 patch_prober.go:28] interesting pod/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 29 21:22:47.326728 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:47.326323 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 29 21:22:48.329585 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:48.329559 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:48.330068 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:48.329830 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:48.330548 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:48.330529 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:22:59.959070 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:59.959036 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz"] Jul 29 21:22:59.961799 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:59.961780 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz" Jul 29 21:22:59.964766 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:59.964744 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:22:59.964766 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:59.964752 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\"" Jul 29 21:22:59.971633 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:22:59.971598 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz"] Jul 29 21:23:00.117549 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:00.117509 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81588fe-e3ba-4921-ad78-9fa0b3f4fefc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz\" (UID: \"a81588fe-e3ba-4921-ad78-9fa0b3f4fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz" Jul 29 21:23:00.218932 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:00.218831 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81588fe-e3ba-4921-ad78-9fa0b3f4fefc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz\" (UID: \"a81588fe-e3ba-4921-ad78-9fa0b3f4fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz" Jul 29 21:23:00.219516 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:00.219493 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81588fe-e3ba-4921-ad78-9fa0b3f4fefc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz\" (UID: \"a81588fe-e3ba-4921-ad78-9fa0b3f4fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz" Jul 29 21:23:00.273763 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:00.273721 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz" Jul 29 21:23:00.413320 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:00.413276 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz"] Jul 29 21:23:00.415564 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:23:00.415539 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81588fe_e3ba_4921_ad78_9fa0b3f4fefc.slice/crio-dc06e408ad7699d0cd67aa2d75ba62de75399a0dd6f9680fca074acf4c0988a4 WatchSource:0}: Error finding container dc06e408ad7699d0cd67aa2d75ba62de75399a0dd6f9680fca074acf4c0988a4: Status 404 returned error can't find the container with id dc06e408ad7699d0cd67aa2d75ba62de75399a0dd6f9680fca074acf4c0988a4 Jul 29 21:23:00.755567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:00.755528 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz" event={"ID":"a81588fe-e3ba-4921-ad78-9fa0b3f4fefc","Type":"ContainerStarted","Data":"3b93af0065a191607b03a14f0a3766eea9b53fe555863a7563a8824d68c170ad"} Jul 29 21:23:00.755753 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:00.755594 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz" event={"ID":"a81588fe-e3ba-4921-ad78-9fa0b3f4fefc","Type":"ContainerStarted","Data":"dc06e408ad7699d0cd67aa2d75ba62de75399a0dd6f9680fca074acf4c0988a4"} Jul 29 21:23:00.772759 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:00.772708 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz" podStartSLOduration=1.7726927350000001 podStartE2EDuration="1.772692735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:23:00.771568247 +0000 UTC m=+2128.895158089" watchObservedRunningTime="2026-07-29 21:23:00.772692735 +0000 UTC m=+2128.896282577" Jul 29 21:23:01.763720 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:01.763681 2582 generic.go:356] "Generic (PLEG): container finished" podID="a81588fe-e3ba-4921-ad78-9fa0b3f4fefc" containerID="3b93af0065a191607b03a14f0a3766eea9b53fe555863a7563a8824d68c170ad" exitCode=0 Jul 29 21:23:01.764159 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:01.763719 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz" event={"ID":"a81588fe-e3ba-4921-ad78-9fa0b3f4fefc","Type":"ContainerDied","Data":"3b93af0065a191607b03a14f0a3766eea9b53fe555863a7563a8824d68c170ad"} Jul 29 21:23:02.905265 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:02.905242 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz" Jul 29 21:23:02.938368 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:02.938337 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81588fe-e3ba-4921-ad78-9fa0b3f4fefc-maas-api-service-ca\") pod \"a81588fe-e3ba-4921-ad78-9fa0b3f4fefc\" (UID: \"a81588fe-e3ba-4921-ad78-9fa0b3f4fefc\") " Jul 29 21:23:02.938781 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:02.938751 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81588fe-e3ba-4921-ad78-9fa0b3f4fefc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a81588fe-e3ba-4921-ad78-9fa0b3f4fefc" (UID: "a81588fe-e3ba-4921-ad78-9fa0b3f4fefc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:23:03.039537 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:03.039435 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81588fe-e3ba-4921-ad78-9fa0b3f4fefc-maas-api-service-ca\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:03.775696 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:03.775659 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz" Jul 29 21:23:03.775696 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:03.775671 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz" event={"ID":"a81588fe-e3ba-4921-ad78-9fa0b3f4fefc","Type":"ContainerDied","Data":"dc06e408ad7699d0cd67aa2d75ba62de75399a0dd6f9680fca074acf4c0988a4"} Jul 29 21:23:03.775961 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:03.775723 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc06e408ad7699d0cd67aa2d75ba62de75399a0dd6f9680fca074acf4c0988a4" Jul 29 21:23:10.588137 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.588086 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp"] Jul 29 21:23:10.589150 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.589112 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" podUID="02748163-23e7-4eb4-847c-709476f093e1" containerName="payload-processing" containerID="cri-o://e5e295f85c97dd94696d2406439e5f3fa8d72f84f29d578589091f881d9c4854" gracePeriod=30 Jul 29 21:23:10.596487 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.596456 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85"] Jul 29 21:23:10.596962 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.596891 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" podUID="190fb518-f280-40e1-8979-aed9540e794c" containerName="payload-pre-processing" containerID="cri-o://9e13fa70f87da0427a27bb18e86a57186608af2f3803b7e4b46d0abcf72457aa" gracePeriod=30 Jul 29 21:23:10.788244 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.788203 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz"] Jul 29 21:23:10.792815 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.792764 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccbf66ba-phjnz"] Jul 29 21:23:10.816143 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.816114 2582 generic.go:356] "Generic (PLEG): container finished" podID="02748163-23e7-4eb4-847c-709476f093e1" containerID="e5e295f85c97dd94696d2406439e5f3fa8d72f84f29d578589091f881d9c4854" exitCode=0 Jul 29 21:23:10.816280 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.816186 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" event={"ID":"02748163-23e7-4eb4-847c-709476f093e1","Type":"ContainerDied","Data":"e5e295f85c97dd94696d2406439e5f3fa8d72f84f29d578589091f881d9c4854"} Jul 29 21:23:10.817701 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.817677 2582 generic.go:356] "Generic (PLEG): container finished" podID="190fb518-f280-40e1-8979-aed9540e794c" containerID="9e13fa70f87da0427a27bb18e86a57186608af2f3803b7e4b46d0abcf72457aa" exitCode=0 Jul 29 21:23:10.817793 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.817741 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" event={"ID":"190fb518-f280-40e1-8979-aed9540e794c","Type":"ContainerDied","Data":"9e13fa70f87da0427a27bb18e86a57186608af2f3803b7e4b46d0abcf72457aa"} Jul 29 21:23:10.857411 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.857386 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" Jul 29 21:23:10.887858 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.887827 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" Jul 29 21:23:10.901366 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.901331 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02748163-23e7-4eb4-847c-709476f093e1-plugins-config-volume\") pod \"02748163-23e7-4eb4-847c-709476f093e1\" (UID: \"02748163-23e7-4eb4-847c-709476f093e1\") " Jul 29 21:23:10.901522 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.901431 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190fb518-f280-40e1-8979-aed9540e794c-plugins-config-volume\") pod \"190fb518-f280-40e1-8979-aed9540e794c\" (UID: \"190fb518-f280-40e1-8979-aed9540e794c\") " Jul 29 21:23:10.901522 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.901513 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d88sx\" (UniqueName: \"kubernetes.io/projected/02748163-23e7-4eb4-847c-709476f093e1-kube-api-access-d88sx\") pod \"02748163-23e7-4eb4-847c-709476f093e1\" (UID: \"02748163-23e7-4eb4-847c-709476f093e1\") " Jul 29 21:23:10.901608 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.901547 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8jsr\" (UniqueName: \"kubernetes.io/projected/190fb518-f280-40e1-8979-aed9540e794c-kube-api-access-h8jsr\") pod \"190fb518-f280-40e1-8979-aed9540e794c\" (UID: \"190fb518-f280-40e1-8979-aed9540e794c\") " Jul 29 21:23:10.901774 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.901746 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02748163-23e7-4eb4-847c-709476f093e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02748163-23e7-4eb4-847c-709476f093e1" (UID: "02748163-23e7-4eb4-847c-709476f093e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:23:10.901816 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.901779 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190fb518-f280-40e1-8979-aed9540e794c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "190fb518-f280-40e1-8979-aed9540e794c" (UID: "190fb518-f280-40e1-8979-aed9540e794c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:23:10.901884 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.901867 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02748163-23e7-4eb4-847c-709476f093e1-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:10.901929 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.901892 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190fb518-f280-40e1-8979-aed9540e794c-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:10.903913 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.903888 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190fb518-f280-40e1-8979-aed9540e794c-kube-api-access-h8jsr" (OuterVolumeSpecName: "kube-api-access-h8jsr") pod "190fb518-f280-40e1-8979-aed9540e794c" (UID: "190fb518-f280-40e1-8979-aed9540e794c"). InnerVolumeSpecName "kube-api-access-h8jsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:23:10.904160 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:10.904135 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02748163-23e7-4eb4-847c-709476f093e1-kube-api-access-d88sx" (OuterVolumeSpecName: "kube-api-access-d88sx") pod "02748163-23e7-4eb4-847c-709476f093e1" (UID: "02748163-23e7-4eb4-847c-709476f093e1"). InnerVolumeSpecName "kube-api-access-d88sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:23:11.002870 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:11.002830 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8jsr\" (UniqueName: \"kubernetes.io/projected/190fb518-f280-40e1-8979-aed9540e794c-kube-api-access-h8jsr\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:11.002870 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:11.002864 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d88sx\" (UniqueName: \"kubernetes.io/projected/02748163-23e7-4eb4-847c-709476f093e1-kube-api-access-d88sx\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:11.824554 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:11.824522 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" Jul 29 21:23:11.825050 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:11.824558 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp" event={"ID":"02748163-23e7-4eb4-847c-709476f093e1","Type":"ContainerDied","Data":"fa1534f781d9c79202aef7f781cdc629cb55e25486d05b9fa6f4b077a92194f2"} Jul 29 21:23:11.825050 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:11.824625 2582 scope.go:117] "RemoveContainer" containerID="e5e295f85c97dd94696d2406439e5f3fa8d72f84f29d578589091f881d9c4854" Jul 29 21:23:11.826108 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:11.826090 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" Jul 29 21:23:11.826397 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:11.826083 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85" event={"ID":"190fb518-f280-40e1-8979-aed9540e794c","Type":"ContainerDied","Data":"595f0a1f85e21d564f844d52430e9a2d971e196f6597ce8bfe869b0829a5a964"} Jul 29 21:23:11.834983 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:11.834967 2582 scope.go:117] "RemoveContainer" containerID="9e13fa70f87da0427a27bb18e86a57186608af2f3803b7e4b46d0abcf72457aa" Jul 29 21:23:11.855063 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:11.855039 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp"] Jul 29 21:23:11.859002 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:11.858979 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccbf66ba-65d7d4f458-hnpwp"] Jul 29 21:23:11.874771 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:11.874710 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85"] Jul 29 21:23:11.877546 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:11.877518 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccbf66ba-67c5548bbd-4kj85"] Jul 29 21:23:12.493441 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:12.493405 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02748163-23e7-4eb4-847c-709476f093e1" path="/var/lib/kubelet/pods/02748163-23e7-4eb4-847c-709476f093e1/volumes" Jul 29 21:23:12.493932 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:12.493915 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="190fb518-f280-40e1-8979-aed9540e794c" path="/var/lib/kubelet/pods/190fb518-f280-40e1-8979-aed9540e794c/volumes" Jul 29 21:23:12.494373 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:12.494357 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a81588fe-e3ba-4921-ad78-9fa0b3f4fefc" path="/var/lib/kubelet/pods/a81588fe-e3ba-4921-ad78-9fa0b3f4fefc/volumes" Jul 29 21:23:13.315804 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:13.315773 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:23:18.657758 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:18.657718 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58"] Jul 29 21:23:18.658291 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:18.657927 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerName="istio-proxy" containerID="cri-o://e229cb4efbe148f6acc02cfec9434d2a98dd1e1ccb246d92089971f97b7e6f2f" gracePeriod=30 Jul 29 21:23:23.894018 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:23.893983 2582 generic.go:356] "Generic (PLEG): container finished" podID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerID="e229cb4efbe148f6acc02cfec9434d2a98dd1e1ccb246d92089971f97b7e6f2f" exitCode=0 Jul 29 21:23:23.894529 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:23.894062 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" event={"ID":"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db","Type":"ContainerDied","Data":"e229cb4efbe148f6acc02cfec9434d2a98dd1e1ccb246d92089971f97b7e6f2f"} Jul 29 21:23:23.894529 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:23.894148 2582 scope.go:117] "RemoveContainer" containerID="a59155714ea75c1bcdfefc4e1a8b6e8c8377bb5160170f4c18114de0cff83d62" Jul 29 21:23:23.927763 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:23.927699 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:23:24.027208 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027150 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-podinfo\") pod \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " Jul 29 21:23:24.027413 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027236 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-workload-socket\") pod \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " Jul 29 21:23:24.027413 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027262 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-envoy\") pod \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " Jul 29 21:23:24.027413 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027281 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-data\") pod \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " Jul 29 21:23:24.027413 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027306 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-token\") pod \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " Jul 29 21:23:24.027413 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027339 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-credential-socket\") pod \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " Jul 29 21:23:24.027668 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027422 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istiod-ca-cert\") pod \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " Jul 29 21:23:24.027668 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027445 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-workload-certs\") pod \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " Jul 29 21:23:24.027668 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027475 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rcm8\" (UniqueName: \"kubernetes.io/projected/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-kube-api-access-9rcm8\") pod \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\" (UID: \"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db\") " Jul 29 21:23:24.027668 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027511 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" (UID: "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:23:24.027865 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027683 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" (UID: "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:23:24.027865 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027776 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" (UID: "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:23:24.027865 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027838 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:24.027865 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027858 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:24.028107 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027874 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:24.028107 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.027952 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" (UID: "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:23:24.028729 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.028698 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-data" (OuterVolumeSpecName: "istio-data") pod "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" (UID: "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:23:24.029827 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.029799 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" (UID: "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:23:24.029928 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.029830 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" (UID: "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:23:24.029928 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.029846 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-kube-api-access-9rcm8" (OuterVolumeSpecName: "kube-api-access-9rcm8") pod "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" (UID: "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db"). InnerVolumeSpecName "kube-api-access-9rcm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:23:24.030087 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.030073 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-token" (OuterVolumeSpecName: "istio-token") pod "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" (UID: "95f20ed4-21e1-4f67-ad2e-c641a3e6a3db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:23:24.129090 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.128997 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:24.129090 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.129076 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:24.129090 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.129085 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:24.129090 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.129095 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:24.129090 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.129104 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rcm8\" (UniqueName: \"kubernetes.io/projected/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-kube-api-access-9rcm8\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:24.129090 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.129112 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:23:24.900736 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.900651 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" event={"ID":"95f20ed4-21e1-4f67-ad2e-c641a3e6a3db","Type":"ContainerDied","Data":"88acea9fda6edd182c78ad637866380a23885d98df9e96a176fb93db54b3f346"} Jul 29 21:23:24.900736 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.900711 2582 scope.go:117] "RemoveContainer" containerID="e229cb4efbe148f6acc02cfec9434d2a98dd1e1ccb246d92089971f97b7e6f2f" Jul 29 21:23:24.901322 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.900659 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58" Jul 29 21:23:24.922004 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.921972 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58"] Jul 29 21:23:24.926655 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:24.926632 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ccbf66ba-openshift-default-65ff56f8d4-lxk58"] Jul 29 21:23:26.492795 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:26.492760 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" path="/var/lib/kubelet/pods/95f20ed4-21e1-4f67-ad2e-c641a3e6a3db/volumes" Jul 29 21:23:38.132369 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132329 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h"] Jul 29 21:23:38.132760 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132696 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190fb518-f280-40e1-8979-aed9540e794c" containerName="payload-pre-processing" Jul 29 21:23:38.132760 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132707 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="190fb518-f280-40e1-8979-aed9540e794c" containerName="payload-pre-processing" Jul 29 21:23:38.132760 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132722 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerName="istio-proxy" Jul 29 21:23:38.132760 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132728 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerName="istio-proxy" Jul 29 21:23:38.132760 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132734 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81588fe-e3ba-4921-ad78-9fa0b3f4fefc" containerName="revoke-keys" Jul 29 21:23:38.132760 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132739 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81588fe-e3ba-4921-ad78-9fa0b3f4fefc" containerName="revoke-keys" Jul 29 21:23:38.132760 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132747 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02748163-23e7-4eb4-847c-709476f093e1" containerName="payload-processing" Jul 29 21:23:38.132760 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132755 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="02748163-23e7-4eb4-847c-709476f093e1" containerName="payload-processing" Jul 29 21:23:38.133041 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132773 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerName="istio-proxy" Jul 29 21:23:38.133041 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132780 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerName="istio-proxy" Jul 29 21:23:38.133041 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132830 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="190fb518-f280-40e1-8979-aed9540e794c" containerName="payload-pre-processing" Jul 29 21:23:38.133041 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132843 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="02748163-23e7-4eb4-847c-709476f093e1" containerName="payload-processing" Jul 29 21:23:38.133041 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132850 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerName="istio-proxy" Jul 29 21:23:38.133041 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.132856 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a81588fe-e3ba-4921-ad78-9fa0b3f4fefc" containerName="revoke-keys" Jul 29 21:23:38.135922 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.135902 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.138548 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.138528 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ac04a8f1-openshift-default-dockercfg-2thp5\"" Jul 29 21:23:38.155751 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.155715 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h"] Jul 29 21:23:38.261206 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.261169 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-envoy\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.261372 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.261219 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-podinfo\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.261372 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.261238 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-data\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.261372 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.261298 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istiod-ca-cert\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.261509 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.261379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-credential-socket\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.261509 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.261405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-workload-certs\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.261509 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.261443 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-workload-socket\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.261509 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.261460 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-token\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.261509 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.261478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh5q6\" (UniqueName: \"kubernetes.io/projected/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-kube-api-access-jh5q6\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.362522 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.362483 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-workload-socket\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.362522 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.362521 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-token\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.362748 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.362541 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh5q6\" (UniqueName: \"kubernetes.io/projected/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-kube-api-access-jh5q6\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.362748 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.362671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-envoy\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.362843 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.362746 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-podinfo\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.362843 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.362770 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-data\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.362843 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.362793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istiod-ca-cert\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.363002 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.362852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-credential-socket\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.363002 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.362895 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-workload-socket\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.363002 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.362916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-workload-certs\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.363200 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.363172 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-data\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.363200 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.363190 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-credential-socket\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.363299 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.363281 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-workload-certs\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.363552 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.363533 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istiod-ca-cert\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.365046 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.365012 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-envoy\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.365377 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.365359 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-podinfo\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.371553 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.371531 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-token\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.371553 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.371539 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh5q6\" (UniqueName: \"kubernetes.io/projected/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-kube-api-access-jh5q6\") pod \"e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.445970 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.445888 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:38.593396 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.593366 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h"] Jul 29 21:23:38.596151 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:23:38.596113 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002f1fcb_aef8_4e47_83c5_d863ffe03c0a.slice/crio-6e145ad37c29c0acec9d0ccd680a7555b0019e401d141b95cd2fd191d7e2ea08 WatchSource:0}: Error finding container 6e145ad37c29c0acec9d0ccd680a7555b0019e401d141b95cd2fd191d7e2ea08: Status 404 returned error can't find the container with id 6e145ad37c29c0acec9d0ccd680a7555b0019e401d141b95cd2fd191d7e2ea08 Jul 29 21:23:38.598707 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.598664 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:23:38.598806 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.598757 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:23:38.598873 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.598802 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:23:38.978107 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.978073 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" event={"ID":"002f1fcb-aef8-4e47-83c5-d863ffe03c0a","Type":"ContainerStarted","Data":"02b8e6dde906cf5c889b34c3f2ede0d30534f6f03b06c1718e7cc57ab2e6e3ae"} Jul 29 21:23:38.978288 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:38.978124 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" event={"ID":"002f1fcb-aef8-4e47-83c5-d863ffe03c0a","Type":"ContainerStarted","Data":"6e145ad37c29c0acec9d0ccd680a7555b0019e401d141b95cd2fd191d7e2ea08"} Jul 29 21:23:39.000890 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:39.000832 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" podStartSLOduration=1.00081283 podStartE2EDuration="1.00081283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:23:38.998966854 +0000 UTC m=+2167.122556695" watchObservedRunningTime="2026-07-29 21:23:39.00081283 +0000 UTC m=+2167.124402672" Jul 29 21:23:39.446869 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:39.446831 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:39.448845 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:39.448820 2582 patch_prober.go:28] interesting pod/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 29 21:23:39.448989 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:39.448868 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 29 21:23:40.451100 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:40.451057 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:40.988319 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:40.988286 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:40.989327 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:40.989307 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:42.832297 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:42.832262 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5"] Jul 29 21:23:42.832780 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:42.832658 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f20ed4-21e1-4f67-ad2e-c641a3e6a3db" containerName="istio-proxy" Jul 29 21:23:42.835712 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:42.835695 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" Jul 29 21:23:42.838691 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:42.838665 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ac04a8f1-dockercfg-kkv42\"" Jul 29 21:23:42.838820 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:42.838701 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ac04a8f1\"" Jul 29 21:23:42.847908 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:42.847883 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5"] Jul 29 21:23:42.859527 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:42.859500 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw"] Jul 29 21:23:42.862802 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:42.862786 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" Jul 29 21:23:42.873344 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:42.873320 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw"] Jul 29 21:23:43.004169 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.004133 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4b6733-5cad-4b3d-8067-3d75ad72eaab-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5\" (UID: \"5a4b6733-5cad-4b3d-8067-3d75ad72eaab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" Jul 29 21:23:43.004346 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.004237 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95-plugins-config-volume\") pod \"payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw\" (UID: \"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95\") " pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" Jul 29 21:23:43.004346 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.004283 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbb6c\" (UniqueName: \"kubernetes.io/projected/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95-kube-api-access-lbb6c\") pod \"payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw\" (UID: \"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95\") " pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" Jul 29 21:23:43.004346 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.004302 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gz45\" (UniqueName: \"kubernetes.io/projected/5a4b6733-5cad-4b3d-8067-3d75ad72eaab-kube-api-access-7gz45\") pod \"payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5\" (UID: \"5a4b6733-5cad-4b3d-8067-3d75ad72eaab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" Jul 29 21:23:43.105893 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.105802 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95-plugins-config-volume\") pod \"payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw\" (UID: \"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95\") " pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" Jul 29 21:23:43.106088 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.105999 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbb6c\" (UniqueName: \"kubernetes.io/projected/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95-kube-api-access-lbb6c\") pod \"payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw\" (UID: \"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95\") " pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" Jul 29 21:23:43.106159 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.106080 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gz45\" (UniqueName: \"kubernetes.io/projected/5a4b6733-5cad-4b3d-8067-3d75ad72eaab-kube-api-access-7gz45\") pod \"payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5\" (UID: \"5a4b6733-5cad-4b3d-8067-3d75ad72eaab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" Jul 29 21:23:43.106227 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.106209 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4b6733-5cad-4b3d-8067-3d75ad72eaab-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5\" (UID: \"5a4b6733-5cad-4b3d-8067-3d75ad72eaab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" Jul 29 21:23:43.106841 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.106817 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95-plugins-config-volume\") pod \"payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw\" (UID: \"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95\") " pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" Jul 29 21:23:43.106898 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.106883 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4b6733-5cad-4b3d-8067-3d75ad72eaab-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5\" (UID: \"5a4b6733-5cad-4b3d-8067-3d75ad72eaab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" Jul 29 21:23:43.115193 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.115170 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gz45\" (UniqueName: \"kubernetes.io/projected/5a4b6733-5cad-4b3d-8067-3d75ad72eaab-kube-api-access-7gz45\") pod \"payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5\" (UID: \"5a4b6733-5cad-4b3d-8067-3d75ad72eaab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" Jul 29 21:23:43.115193 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.115184 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbb6c\" (UniqueName: \"kubernetes.io/projected/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95-kube-api-access-lbb6c\") pod \"payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw\" (UID: \"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95\") " pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" Jul 29 21:23:43.147919 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.147882 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" Jul 29 21:23:43.174231 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.174202 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" Jul 29 21:23:43.333568 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.333545 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw"] Jul 29 21:23:43.335127 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:23:43.335097 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb9b5d9_e2d8_4568_a992_5e01dbfafa95.slice/crio-bbd8d98cf57a9961404093071a10219008811c86d2d10c2f14195fea5e0f62b9 WatchSource:0}: Error finding container bbd8d98cf57a9961404093071a10219008811c86d2d10c2f14195fea5e0f62b9: Status 404 returned error can't find the container with id bbd8d98cf57a9961404093071a10219008811c86d2d10c2f14195fea5e0f62b9 Jul 29 21:23:43.506097 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:43.506066 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5"] Jul 29 21:23:43.507963 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:23:43.507926 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4b6733_5cad_4b3d_8067_3d75ad72eaab.slice/crio-32c257b7c461ce9fc054e2be86b725843e03d238c9e82aef301177b9e59669ca WatchSource:0}: Error finding container 32c257b7c461ce9fc054e2be86b725843e03d238c9e82aef301177b9e59669ca: Status 404 returned error can't find the container with id 32c257b7c461ce9fc054e2be86b725843e03d238c9e82aef301177b9e59669ca Jul 29 21:23:44.009694 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:44.009658 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" event={"ID":"5a4b6733-5cad-4b3d-8067-3d75ad72eaab","Type":"ContainerStarted","Data":"dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852"} Jul 29 21:23:44.010279 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:44.009724 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" event={"ID":"5a4b6733-5cad-4b3d-8067-3d75ad72eaab","Type":"ContainerStarted","Data":"32c257b7c461ce9fc054e2be86b725843e03d238c9e82aef301177b9e59669ca"} Jul 29 21:23:44.010279 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:44.009835 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" Jul 29 21:23:44.011496 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:44.011463 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" event={"ID":"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95","Type":"ContainerStarted","Data":"005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f"} Jul 29 21:23:44.011606 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:44.011512 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" event={"ID":"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95","Type":"ContainerStarted","Data":"bbd8d98cf57a9961404093071a10219008811c86d2d10c2f14195fea5e0f62b9"} Jul 29 21:23:44.011606 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:44.011571 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" Jul 29 21:23:44.027787 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:44.027728 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" podStartSLOduration=2.027708996 podStartE2EDuration="2.027708996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:23:44.025516509 +0000 UTC m=+2172.149106352" watchObservedRunningTime="2026-07-29 21:23:44.027708996 +0000 UTC m=+2172.151298839" Jul 29 21:23:44.043496 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:44.043446 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" podStartSLOduration=2.043432625 podStartE2EDuration="2.043432625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:23:44.041678858 +0000 UTC m=+2172.165268703" watchObservedRunningTime="2026-07-29 21:23:44.043432625 +0000 UTC m=+2172.167022466" Jul 29 21:23:46.153691 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:46.153650 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:23:54.070679 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:54.070629 2582 generic.go:356] "Generic (PLEG): container finished" podID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerID="02b8e6dde906cf5c889b34c3f2ede0d30534f6f03b06c1718e7cc57ab2e6e3ae" exitCode=0 Jul 29 21:23:54.071131 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:54.070706 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" event={"ID":"002f1fcb-aef8-4e47-83c5-d863ffe03c0a","Type":"ContainerDied","Data":"02b8e6dde906cf5c889b34c3f2ede0d30534f6f03b06c1718e7cc57ab2e6e3ae"} Jul 29 21:23:54.071179 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:54.071130 2582 scope.go:117] "RemoveContainer" containerID="02b8e6dde906cf5c889b34c3f2ede0d30534f6f03b06c1718e7cc57ab2e6e3ae" Jul 29 21:23:54.072207 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:54.072181 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:23:54.072282 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:54.072247 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:23:54.072324 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:54.072279 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:23:55.019225 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:55.019188 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" Jul 29 21:23:55.019789 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:55.019760 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" Jul 29 21:23:55.079001 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:55.078971 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" event={"ID":"002f1fcb-aef8-4e47-83c5-d863ffe03c0a","Type":"ContainerStarted","Data":"eefa1736f0fd1c4dbd9f244750ced19d275634fa6c05f86d7dfeba1375d73682"} Jul 29 21:23:55.446590 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:55.446498 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:55.447846 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:55.447822 2582 patch_prober.go:28] interesting pod/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 29 21:23:55.447968 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:55.447871 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 29 21:23:55.988424 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:55.988389 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:56.446470 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:56.446383 2582 patch_prober.go:28] interesting pod/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 29 21:23:56.446470 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:56.446442 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 29 21:23:57.446740 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:57.446705 2582 patch_prober.go:28] interesting pod/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 29 21:23:57.447169 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:57.446764 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 29 21:23:58.447152 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:58.447114 2582 patch_prober.go:28] interesting pod/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 29 21:23:58.447553 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:58.447179 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 29 21:23:59.450989 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:59.450959 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:23:59.451995 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:23:59.451976 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:24:14.762780 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:14.762741 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:24:31.054382 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:31.054344 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:24:35.222741 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:35.222703 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s"] Jul 29 21:24:35.227729 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:35.227711 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s" Jul 29 21:24:35.230477 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:35.230453 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\"" Jul 29 21:24:35.230612 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:35.230461 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:24:35.234740 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:35.234716 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s"] Jul 29 21:24:35.404359 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:35.404314 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d05784-63ff-4681-9f6c-6f58d9850905-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s\" (UID: \"51d05784-63ff-4681-9f6c-6f58d9850905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s" Jul 29 21:24:35.505458 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:35.505421 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d05784-63ff-4681-9f6c-6f58d9850905-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s\" (UID: \"51d05784-63ff-4681-9f6c-6f58d9850905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s" Jul 29 21:24:35.506093 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:35.506075 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d05784-63ff-4681-9f6c-6f58d9850905-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s\" (UID: \"51d05784-63ff-4681-9f6c-6f58d9850905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s" Jul 29 21:24:35.540387 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:35.540354 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s" Jul 29 21:24:35.685218 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:35.685160 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s"] Jul 29 21:24:35.687587 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:24:35.687554 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d05784_63ff_4681_9f6c_6f58d9850905.slice/crio-74d73e79778e8cc209f7b567ea8df49052d1dac43a62b85c407dafe162731727 WatchSource:0}: Error finding container 74d73e79778e8cc209f7b567ea8df49052d1dac43a62b85c407dafe162731727: Status 404 returned error can't find the container with id 74d73e79778e8cc209f7b567ea8df49052d1dac43a62b85c407dafe162731727 Jul 29 21:24:36.306148 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:36.306109 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s" event={"ID":"51d05784-63ff-4681-9f6c-6f58d9850905","Type":"ContainerStarted","Data":"3d7a2cc86d9dd741030a34e9d1c3fe8a32bd3ac217a8be2cda9d54e2482414e1"} Jul 29 21:24:36.306638 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:36.306164 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s" event={"ID":"51d05784-63ff-4681-9f6c-6f58d9850905","Type":"ContainerStarted","Data":"74d73e79778e8cc209f7b567ea8df49052d1dac43a62b85c407dafe162731727"} Jul 29 21:24:36.322500 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:36.322453 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s" podStartSLOduration=1.322435765 podStartE2EDuration="1.322435765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:24:36.32155601 +0000 UTC m=+2224.445145851" watchObservedRunningTime="2026-07-29 21:24:36.322435765 +0000 UTC m=+2224.446025606" Jul 29 21:24:37.313556 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:37.313526 2582 generic.go:356] "Generic (PLEG): container finished" podID="51d05784-63ff-4681-9f6c-6f58d9850905" containerID="3d7a2cc86d9dd741030a34e9d1c3fe8a32bd3ac217a8be2cda9d54e2482414e1" exitCode=0 Jul 29 21:24:37.313962 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:37.313601 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s" event={"ID":"51d05784-63ff-4681-9f6c-6f58d9850905","Type":"ContainerDied","Data":"3d7a2cc86d9dd741030a34e9d1c3fe8a32bd3ac217a8be2cda9d54e2482414e1"} Jul 29 21:24:38.453180 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:38.453153 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s" Jul 29 21:24:38.533164 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:38.533128 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d05784-63ff-4681-9f6c-6f58d9850905-maas-api-service-ca\") pod \"51d05784-63ff-4681-9f6c-6f58d9850905\" (UID: \"51d05784-63ff-4681-9f6c-6f58d9850905\") " Jul 29 21:24:38.533529 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:38.533503 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d05784-63ff-4681-9f6c-6f58d9850905-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51d05784-63ff-4681-9f6c-6f58d9850905" (UID: "51d05784-63ff-4681-9f6c-6f58d9850905"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:24:38.634128 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:38.634017 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d05784-63ff-4681-9f6c-6f58d9850905-maas-api-service-ca\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:24:39.326351 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:39.326309 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s" event={"ID":"51d05784-63ff-4681-9f6c-6f58d9850905","Type":"ContainerDied","Data":"74d73e79778e8cc209f7b567ea8df49052d1dac43a62b85c407dafe162731727"} Jul 29 21:24:39.326351 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:39.326352 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s" Jul 29 21:24:39.326573 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:39.326376 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74d73e79778e8cc209f7b567ea8df49052d1dac43a62b85c407dafe162731727" Jul 29 21:24:49.269192 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:49.269156 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:24:50.834949 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:50.834899 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw"] Jul 29 21:24:50.835589 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:50.835207 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" podUID="2eb9b5d9-e2d8-4568-a992-5e01dbfafa95" containerName="payload-processing" containerID="cri-o://005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f" gracePeriod=30 Jul 29 21:24:50.847589 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:50.847550 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5"] Jul 29 21:24:50.847837 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:50.847796 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" podUID="5a4b6733-5cad-4b3d-8067-3d75ad72eaab" containerName="payload-pre-processing" containerID="cri-o://dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852" gracePeriod=30 Jul 29 21:24:51.101972 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.101879 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s"] Jul 29 21:24:51.116044 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.116000 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac04a8f1-7qx9s"] Jul 29 21:24:51.137857 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.137825 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" Jul 29 21:24:51.140627 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.140607 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" Jul 29 21:24:51.247840 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.247811 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95-plugins-config-volume\") pod \"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95\" (UID: \"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95\") " Jul 29 21:24:51.248083 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.247850 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gz45\" (UniqueName: \"kubernetes.io/projected/5a4b6733-5cad-4b3d-8067-3d75ad72eaab-kube-api-access-7gz45\") pod \"5a4b6733-5cad-4b3d-8067-3d75ad72eaab\" (UID: \"5a4b6733-5cad-4b3d-8067-3d75ad72eaab\") " Jul 29 21:24:51.248083 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.247909 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbb6c\" (UniqueName: \"kubernetes.io/projected/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95-kube-api-access-lbb6c\") pod \"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95\" (UID: \"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95\") " Jul 29 21:24:51.248083 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.247946 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4b6733-5cad-4b3d-8067-3d75ad72eaab-plugins-config-volume\") pod \"5a4b6733-5cad-4b3d-8067-3d75ad72eaab\" (UID: \"5a4b6733-5cad-4b3d-8067-3d75ad72eaab\") " Jul 29 21:24:51.248357 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.248331 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2eb9b5d9-e2d8-4568-a992-5e01dbfafa95" (UID: "2eb9b5d9-e2d8-4568-a992-5e01dbfafa95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:24:51.248425 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.248372 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4b6733-5cad-4b3d-8067-3d75ad72eaab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a4b6733-5cad-4b3d-8067-3d75ad72eaab" (UID: "5a4b6733-5cad-4b3d-8067-3d75ad72eaab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:24:51.250274 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.250251 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4b6733-5cad-4b3d-8067-3d75ad72eaab-kube-api-access-7gz45" (OuterVolumeSpecName: "kube-api-access-7gz45") pod "5a4b6733-5cad-4b3d-8067-3d75ad72eaab" (UID: "5a4b6733-5cad-4b3d-8067-3d75ad72eaab"). InnerVolumeSpecName "kube-api-access-7gz45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:24:51.250384 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.250280 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95-kube-api-access-lbb6c" (OuterVolumeSpecName: "kube-api-access-lbb6c") pod "2eb9b5d9-e2d8-4568-a992-5e01dbfafa95" (UID: "2eb9b5d9-e2d8-4568-a992-5e01dbfafa95"). InnerVolumeSpecName "kube-api-access-lbb6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:24:51.349402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.349362 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4b6733-5cad-4b3d-8067-3d75ad72eaab-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:24:51.349402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.349396 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:24:51.349402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.349406 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gz45\" (UniqueName: \"kubernetes.io/projected/5a4b6733-5cad-4b3d-8067-3d75ad72eaab-kube-api-access-7gz45\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:24:51.349652 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.349417 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbb6c\" (UniqueName: \"kubernetes.io/projected/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95-kube-api-access-lbb6c\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:24:51.392421 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.392326 2582 generic.go:356] "Generic (PLEG): container finished" podID="5a4b6733-5cad-4b3d-8067-3d75ad72eaab" containerID="dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852" exitCode=0 Jul 29 21:24:51.392595 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.392394 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" event={"ID":"5a4b6733-5cad-4b3d-8067-3d75ad72eaab","Type":"ContainerDied","Data":"dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852"} Jul 29 21:24:51.392595 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.392474 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" event={"ID":"5a4b6733-5cad-4b3d-8067-3d75ad72eaab","Type":"ContainerDied","Data":"32c257b7c461ce9fc054e2be86b725843e03d238c9e82aef301177b9e59669ca"} Jul 29 21:24:51.392595 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.392508 2582 scope.go:117] "RemoveContainer" containerID="dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852" Jul 29 21:24:51.392753 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.392710 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5" Jul 29 21:24:51.395116 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.394724 2582 generic.go:356] "Generic (PLEG): container finished" podID="2eb9b5d9-e2d8-4568-a992-5e01dbfafa95" containerID="005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f" exitCode=0 Jul 29 21:24:51.395116 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.394762 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" event={"ID":"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95","Type":"ContainerDied","Data":"005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f"} Jul 29 21:24:51.395116 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.394807 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" event={"ID":"2eb9b5d9-e2d8-4568-a992-5e01dbfafa95","Type":"ContainerDied","Data":"bbd8d98cf57a9961404093071a10219008811c86d2d10c2f14195fea5e0f62b9"} Jul 29 21:24:51.395116 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.394887 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw" Jul 29 21:24:51.402868 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.402849 2582 scope.go:117] "RemoveContainer" containerID="dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852" Jul 29 21:24:51.403217 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:24:51.403190 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852\": container with ID starting with dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852 not found: ID does not exist" containerID="dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852" Jul 29 21:24:51.403453 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.403221 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852"} err="failed to get container status \"dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852\": rpc error: code = NotFound desc = could not find container \"dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852\": container with ID starting with dda4882a17a29736035624a3ba41fbb31924328998a94cd41cb970c859c62852 not found: ID does not exist" Jul 29 21:24:51.403453 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.403237 2582 scope.go:117] "RemoveContainer" containerID="005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f" Jul 29 21:24:51.411996 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.411978 2582 scope.go:117] "RemoveContainer" containerID="005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f" Jul 29 21:24:51.412280 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:24:51.412261 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f\": container with ID starting with 005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f not found: ID does not exist" containerID="005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f" Jul 29 21:24:51.412339 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.412287 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f"} err="failed to get container status \"005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f\": rpc error: code = NotFound desc = could not find container \"005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f\": container with ID starting with 005f430a3c98454f15bf424ae3c073c6c95e790f24a87445c44a9d4f4a83836f not found: ID does not exist" Jul 29 21:24:51.420680 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.420649 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5"] Jul 29 21:24:51.427592 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.427566 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ac04a8f1-9ffcd5f9b-ll7t5"] Jul 29 21:24:51.447714 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.447680 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw"] Jul 29 21:24:51.449431 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:51.449413 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ac04a8f1-8dcff56db-c7mnw"] Jul 29 21:24:52.492859 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:52.492825 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eb9b5d9-e2d8-4568-a992-5e01dbfafa95" path="/var/lib/kubelet/pods/2eb9b5d9-e2d8-4568-a992-5e01dbfafa95/volumes" Jul 29 21:24:52.493367 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:52.493212 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51d05784-63ff-4681-9f6c-6f58d9850905" path="/var/lib/kubelet/pods/51d05784-63ff-4681-9f6c-6f58d9850905/volumes" Jul 29 21:24:52.493551 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:24:52.493537 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4b6733-5cad-4b3d-8067-3d75ad72eaab" path="/var/lib/kubelet/pods/5a4b6733-5cad-4b3d-8067-3d75ad72eaab/volumes" Jul 29 21:25:00.014095 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:00.013990 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:25:32.632350 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:32.632315 2582 scope.go:117] "RemoveContainer" containerID="b2e61e6ff96dbbe22d4ab75535710e839fd9337855c373d0fe89566cf50d0c41" Jul 29 21:25:32.641097 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:32.641075 2582 scope.go:117] "RemoveContainer" containerID="13c09d33e6bd12a4ac807aa44af2a6dfb2d82169e300ec3c8c6969cc3a36779b" Jul 29 21:25:43.574850 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:43.574818 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h"] Jul 29 21:25:43.575365 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:43.575200 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" containerID="cri-o://eefa1736f0fd1c4dbd9f244750ced19d275634fa6c05f86d7dfeba1375d73682" gracePeriod=30 Jul 29 21:25:44.454098 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:44.454058 2582 patch_prober.go:28] interesting pod/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 21:25:44.454278 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:44.454117 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 21:25:47.168635 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:47.168592 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:25:48.709227 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.709190 2582 generic.go:356] "Generic (PLEG): container finished" podID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerID="eefa1736f0fd1c4dbd9f244750ced19d275634fa6c05f86d7dfeba1375d73682" exitCode=0 Jul 29 21:25:48.709660 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.709244 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" event={"ID":"002f1fcb-aef8-4e47-83c5-d863ffe03c0a","Type":"ContainerDied","Data":"eefa1736f0fd1c4dbd9f244750ced19d275634fa6c05f86d7dfeba1375d73682"} Jul 29 21:25:48.709660 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.709308 2582 scope.go:117] "RemoveContainer" containerID="02b8e6dde906cf5c889b34c3f2ede0d30534f6f03b06c1718e7cc57ab2e6e3ae" Jul 29 21:25:48.838947 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.838884 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:25:48.963804 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.963774 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-envoy\") pod \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " Jul 29 21:25:48.963995 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.963869 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-workload-certs\") pod \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " Jul 29 21:25:48.963995 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.963890 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-data\") pod \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " Jul 29 21:25:48.963995 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.963904 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-workload-socket\") pod \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " Jul 29 21:25:48.963995 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.963928 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istiod-ca-cert\") pod \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " Jul 29 21:25:48.963995 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.963943 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-credential-socket\") pod \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " Jul 29 21:25:48.963995 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.963962 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh5q6\" (UniqueName: \"kubernetes.io/projected/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-kube-api-access-jh5q6\") pod \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " Jul 29 21:25:48.964295 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.964008 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-token\") pod \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " Jul 29 21:25:48.964295 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.964070 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-podinfo\") pod \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\" (UID: \"002f1fcb-aef8-4e47-83c5-d863ffe03c0a\") " Jul 29 21:25:48.964295 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.964186 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "002f1fcb-aef8-4e47-83c5-d863ffe03c0a" (UID: "002f1fcb-aef8-4e47-83c5-d863ffe03c0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:25:48.964456 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.964408 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:25:48.964456 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.964403 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "002f1fcb-aef8-4e47-83c5-d863ffe03c0a" (UID: "002f1fcb-aef8-4e47-83c5-d863ffe03c0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:25:48.964550 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.964481 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "002f1fcb-aef8-4e47-83c5-d863ffe03c0a" (UID: "002f1fcb-aef8-4e47-83c5-d863ffe03c0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:25:48.964871 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.964732 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "002f1fcb-aef8-4e47-83c5-d863ffe03c0a" (UID: "002f1fcb-aef8-4e47-83c5-d863ffe03c0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:25:48.964871 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.964802 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-data" (OuterVolumeSpecName: "istio-data") pod "002f1fcb-aef8-4e47-83c5-d863ffe03c0a" (UID: "002f1fcb-aef8-4e47-83c5-d863ffe03c0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:25:48.966654 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.966620 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-token" (OuterVolumeSpecName: "istio-token") pod "002f1fcb-aef8-4e47-83c5-d863ffe03c0a" (UID: "002f1fcb-aef8-4e47-83c5-d863ffe03c0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:25:48.966766 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.966664 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "002f1fcb-aef8-4e47-83c5-d863ffe03c0a" (UID: "002f1fcb-aef8-4e47-83c5-d863ffe03c0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:25:48.966766 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.966746 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "002f1fcb-aef8-4e47-83c5-d863ffe03c0a" (UID: "002f1fcb-aef8-4e47-83c5-d863ffe03c0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:25:48.966918 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:48.966880 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-kube-api-access-jh5q6" (OuterVolumeSpecName: "kube-api-access-jh5q6") pod "002f1fcb-aef8-4e47-83c5-d863ffe03c0a" (UID: "002f1fcb-aef8-4e47-83c5-d863ffe03c0a"). InnerVolumeSpecName "kube-api-access-jh5q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:25:49.065662 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.065617 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:25:49.065662 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.065663 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:25:49.065905 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.065680 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh5q6\" (UniqueName: \"kubernetes.io/projected/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-kube-api-access-jh5q6\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:25:49.065905 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.065696 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:25:49.065905 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.065711 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:25:49.065905 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.065722 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:25:49.065905 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.065735 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:25:49.065905 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.065747 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002f1fcb-aef8-4e47-83c5-d863ffe03c0a-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:25:49.716564 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.716522 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" Jul 29 21:25:49.717089 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.716522 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h" event={"ID":"002f1fcb-aef8-4e47-83c5-d863ffe03c0a","Type":"ContainerDied","Data":"6e145ad37c29c0acec9d0ccd680a7555b0019e401d141b95cd2fd191d7e2ea08"} Jul 29 21:25:49.717089 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.716674 2582 scope.go:117] "RemoveContainer" containerID="eefa1736f0fd1c4dbd9f244750ced19d275634fa6c05f86d7dfeba1375d73682" Jul 29 21:25:49.742326 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.742294 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h"] Jul 29 21:25:49.747722 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:49.747694 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ac04a8f1-openshift-default-5469cb4b48-k9m2h"] Jul 29 21:25:50.493918 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:50.493872 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" path="/var/lib/kubelet/pods/002f1fcb-aef8-4e47-83c5-d863ffe03c0a/volumes" Jul 29 21:25:57.764002 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:25:57.763960 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:26:32.674324 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:32.674248 2582 scope.go:117] "RemoveContainer" containerID="a0a6770a46e0a0eb285a959557af145da94c7e0658d56bb9a76b77d39524ab98" Jul 29 21:26:32.682749 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:32.682730 2582 scope.go:117] "RemoveContainer" containerID="120c23b09a47c1818dcbae939246cd0f720c3bc8f9737448975a074de17d9fd0" Jul 29 21:26:41.395062 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395010 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv"] Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395366 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4b6733-5cad-4b3d-8067-3d75ad72eaab" containerName="payload-pre-processing" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395378 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4b6733-5cad-4b3d-8067-3d75ad72eaab" containerName="payload-pre-processing" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395414 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb9b5d9-e2d8-4568-a992-5e01dbfafa95" containerName="payload-processing" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395421 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb9b5d9-e2d8-4568-a992-5e01dbfafa95" containerName="payload-processing" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395430 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51d05784-63ff-4681-9f6c-6f58d9850905" containerName="revoke-keys" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395435 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="51d05784-63ff-4681-9f6c-6f58d9850905" containerName="revoke-keys" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395445 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395451 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395458 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395463 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395510 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395516 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb9b5d9-e2d8-4568-a992-5e01dbfafa95" containerName="payload-processing" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395524 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4b6733-5cad-4b3d-8067-3d75ad72eaab" containerName="payload-pre-processing" Jul 29 21:26:41.395567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.395530 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="51d05784-63ff-4681-9f6c-6f58d9850905" containerName="revoke-keys" Jul 29 21:26:41.398447 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.398425 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.402228 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.402201 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4d12bd-openshift-default-dockercfg-lwhwk\"" Jul 29 21:26:41.411136 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.411108 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv"] Jul 29 21:26:41.556506 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.556459 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-workload-certs\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.556506 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.556509 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-data\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.556726 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.556625 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istiod-ca-cert\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.556726 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.556658 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qv2\" (UniqueName: \"kubernetes.io/projected/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-kube-api-access-b9qv2\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.556726 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.556690 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-credential-socket\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.556726 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.556716 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-envoy\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.556864 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.556739 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-token\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.556864 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.556768 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-podinfo\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.556864 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.556803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-workload-socket\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.658093 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.657956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-envoy\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.658093 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.657998 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-token\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.658354 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.658174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-podinfo\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.658354 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.658242 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-workload-socket\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.658354 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.658355 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-workload-certs\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.658551 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.658394 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-data\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.658551 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.658468 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istiod-ca-cert\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.658551 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.658496 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qv2\" (UniqueName: \"kubernetes.io/projected/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-kube-api-access-b9qv2\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.658551 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.658541 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-credential-socket\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.658758 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.658624 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-workload-socket\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.658758 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.658723 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-workload-certs\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.659315 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.658799 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-data\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.659315 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.658859 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-credential-socket\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.659315 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.659221 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istiod-ca-cert\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.660587 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.660566 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-envoy\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.660944 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.660916 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-podinfo\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.669443 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.669418 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qv2\" (UniqueName: \"kubernetes.io/projected/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-kube-api-access-b9qv2\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.670003 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.669986 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-token\") pod \"e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.709258 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.709221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:41.879212 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.879165 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv"] Jul 29 21:26:41.883754 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:26:41.883726 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a13bfcf_2b73_40bb_8de5_7eb2cd00a38c.slice/crio-cb3121b0f19b0e6a60e8a513e8c65b99235a5006b4d8a990d17bacacc1080a40 WatchSource:0}: Error finding container cb3121b0f19b0e6a60e8a513e8c65b99235a5006b4d8a990d17bacacc1080a40: Status 404 returned error can't find the container with id cb3121b0f19b0e6a60e8a513e8c65b99235a5006b4d8a990d17bacacc1080a40 Jul 29 21:26:41.885806 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.885774 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:26:41.885882 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.885847 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:26:41.885882 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:41.885875 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:26:42.008981 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:42.008933 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" event={"ID":"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c","Type":"ContainerStarted","Data":"4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e"} Jul 29 21:26:42.009196 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:42.009009 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" event={"ID":"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c","Type":"ContainerStarted","Data":"cb3121b0f19b0e6a60e8a513e8c65b99235a5006b4d8a990d17bacacc1080a40"} Jul 29 21:26:42.038367 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:42.038307 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" podStartSLOduration=1.038290733 podStartE2EDuration="1.038290733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:26:42.03672551 +0000 UTC m=+2350.160315352" watchObservedRunningTime="2026-07-29 21:26:42.038290733 +0000 UTC m=+2350.161880574" Jul 29 21:26:42.709940 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:42.709900 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:42.715062 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:42.715005 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:43.014104 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:43.014067 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:43.015093 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:43.015073 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:26:45.865087 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:45.865045 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b"] Jul 29 21:26:45.865669 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:45.865632 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="002f1fcb-aef8-4e47-83c5-d863ffe03c0a" containerName="istio-proxy" Jul 29 21:26:45.868717 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:45.868701 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" Jul 29 21:26:45.871733 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:45.871715 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4d12bd\"" Jul 29 21:26:45.872125 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:45.872108 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4d12bd-dockercfg-bdhrl\"" Jul 29 21:26:45.880054 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:45.880011 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b"] Jul 29 21:26:45.886633 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:45.886606 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9"] Jul 29 21:26:45.890014 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:45.889993 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" Jul 29 21:26:45.911408 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:45.911378 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9"] Jul 29 21:26:46.000869 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.000828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb5391-1db3-4a74-b593-720b28211502-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b\" (UID: \"b1cb5391-1db3-4a74-b593-720b28211502\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" Jul 29 21:26:46.000869 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.000875 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqb46\" (UniqueName: \"kubernetes.io/projected/b1cb5391-1db3-4a74-b593-720b28211502-kube-api-access-kqb46\") pod \"payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b\" (UID: \"b1cb5391-1db3-4a74-b593-720b28211502\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" Jul 29 21:26:46.001167 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.000971 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txr6r\" (UniqueName: \"kubernetes.io/projected/fa858ba9-4961-4a69-9eea-9135eb4ade30-kube-api-access-txr6r\") pod \"payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9\" (UID: \"fa858ba9-4961-4a69-9eea-9135eb4ade30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" Jul 29 21:26:46.001167 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.001101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa858ba9-4961-4a69-9eea-9135eb4ade30-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9\" (UID: \"fa858ba9-4961-4a69-9eea-9135eb4ade30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" Jul 29 21:26:46.102175 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.102132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb5391-1db3-4a74-b593-720b28211502-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b\" (UID: \"b1cb5391-1db3-4a74-b593-720b28211502\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" Jul 29 21:26:46.102395 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.102196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqb46\" (UniqueName: \"kubernetes.io/projected/b1cb5391-1db3-4a74-b593-720b28211502-kube-api-access-kqb46\") pod \"payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b\" (UID: \"b1cb5391-1db3-4a74-b593-720b28211502\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" Jul 29 21:26:46.102395 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.102255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txr6r\" (UniqueName: \"kubernetes.io/projected/fa858ba9-4961-4a69-9eea-9135eb4ade30-kube-api-access-txr6r\") pod \"payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9\" (UID: \"fa858ba9-4961-4a69-9eea-9135eb4ade30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" Jul 29 21:26:46.102395 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.102326 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa858ba9-4961-4a69-9eea-9135eb4ade30-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9\" (UID: \"fa858ba9-4961-4a69-9eea-9135eb4ade30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" Jul 29 21:26:46.102965 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.102940 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb5391-1db3-4a74-b593-720b28211502-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b\" (UID: \"b1cb5391-1db3-4a74-b593-720b28211502\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" Jul 29 21:26:46.103093 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.103072 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa858ba9-4961-4a69-9eea-9135eb4ade30-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9\" (UID: \"fa858ba9-4961-4a69-9eea-9135eb4ade30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" Jul 29 21:26:46.110798 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.110766 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txr6r\" (UniqueName: \"kubernetes.io/projected/fa858ba9-4961-4a69-9eea-9135eb4ade30-kube-api-access-txr6r\") pod \"payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9\" (UID: \"fa858ba9-4961-4a69-9eea-9135eb4ade30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" Jul 29 21:26:46.110910 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.110803 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqb46\" (UniqueName: \"kubernetes.io/projected/b1cb5391-1db3-4a74-b593-720b28211502-kube-api-access-kqb46\") pod \"payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b\" (UID: \"b1cb5391-1db3-4a74-b593-720b28211502\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" Jul 29 21:26:46.179634 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.179534 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" Jul 29 21:26:46.203212 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.203179 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" Jul 29 21:26:46.338950 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.338919 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b"] Jul 29 21:26:46.340429 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:26:46.340398 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb5391_1db3_4a74_b593_720b28211502.slice/crio-6c23b129d497c2fb4c745717f798c240d435e79613f1468e4797dc25245c5c7c WatchSource:0}: Error finding container 6c23b129d497c2fb4c745717f798c240d435e79613f1468e4797dc25245c5c7c: Status 404 returned error can't find the container with id 6c23b129d497c2fb4c745717f798c240d435e79613f1468e4797dc25245c5c7c Jul 29 21:26:46.360332 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:46.360310 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9"] Jul 29 21:26:46.362893 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:26:46.362869 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa858ba9_4961_4a69_9eea_9135eb4ade30.slice/crio-bdca623221c9032409d3ff23d9f63c68e29b1fb051a00c5d3fb401152e578b20 WatchSource:0}: Error finding container bdca623221c9032409d3ff23d9f63c68e29b1fb051a00c5d3fb401152e578b20: Status 404 returned error can't find the container with id bdca623221c9032409d3ff23d9f63c68e29b1fb051a00c5d3fb401152e578b20 Jul 29 21:26:47.037688 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:47.037645 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" event={"ID":"fa858ba9-4961-4a69-9eea-9135eb4ade30","Type":"ContainerStarted","Data":"4dc3fcd22f594679e46df90f56f409453105cde1f30b6d4e9a4361a1afed2004"} Jul 29 21:26:47.038318 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:47.037719 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" Jul 29 21:26:47.038318 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:47.037738 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" event={"ID":"fa858ba9-4961-4a69-9eea-9135eb4ade30","Type":"ContainerStarted","Data":"bdca623221c9032409d3ff23d9f63c68e29b1fb051a00c5d3fb401152e578b20"} Jul 29 21:26:47.039512 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:47.039479 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" event={"ID":"b1cb5391-1db3-4a74-b593-720b28211502","Type":"ContainerStarted","Data":"331bfbc5ab15ce19e04cf940f025e3139f6fbe403b243a343fb1f738001dfd7d"} Jul 29 21:26:47.039642 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:47.039526 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" event={"ID":"b1cb5391-1db3-4a74-b593-720b28211502","Type":"ContainerStarted","Data":"6c23b129d497c2fb4c745717f798c240d435e79613f1468e4797dc25245c5c7c"} Jul 29 21:26:47.039642 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:47.039549 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" Jul 29 21:26:47.071214 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:47.071161 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" podStartSLOduration=2.071141237 podStartE2EDuration="2.071141237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:26:47.062505628 +0000 UTC m=+2355.186095470" watchObservedRunningTime="2026-07-29 21:26:47.071141237 +0000 UTC m=+2355.194731078" Jul 29 21:26:47.093509 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:47.093447 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" podStartSLOduration=2.0934241829999998 podStartE2EDuration="2.093424183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:26:47.09166423 +0000 UTC m=+2355.215254072" watchObservedRunningTime="2026-07-29 21:26:47.093424183 +0000 UTC m=+2355.217014026" Jul 29 21:26:50.548065 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:50.548016 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:26:58.046937 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:58.046907 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" Jul 29 21:26:58.047344 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:58.046985 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" Jul 29 21:26:58.297954 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:58.297923 2582 patch_prober.go:28] interesting pod/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": read tcp 10.134.0.2:56058->10.134.0.72:15021: read: connection reset by peer" start-of-body= Jul 29 21:26:58.298109 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:58.297997 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": read tcp 10.134.0.2:56058->10.134.0.72:15021: read: connection reset by peer" Jul 29 21:26:59.108737 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:59.108704 2582 generic.go:356] "Generic (PLEG): container finished" podID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerID="4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e" exitCode=0 Jul 29 21:26:59.108737 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:59.108742 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" event={"ID":"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c","Type":"ContainerDied","Data":"4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e"} Jul 29 21:26:59.109191 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:59.109149 2582 scope.go:117] "RemoveContainer" containerID="4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e" Jul 29 21:26:59.110178 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:59.110148 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:26:59.110253 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:59.110240 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:26:59.110291 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:26:59.110274 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:27:00.119241 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:00.119198 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" event={"ID":"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c","Type":"ContainerStarted","Data":"e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2"} Jul 29 21:27:00.710205 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:00.710163 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:27:00.710580 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:00.710551 2582 patch_prober.go:28] interesting pod/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 29 21:27:00.710661 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:00.710611 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 29 21:27:01.709850 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:01.709810 2582 patch_prober.go:28] interesting pod/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 29 21:27:01.710522 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:01.710491 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 29 21:27:02.709962 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:02.709915 2582 patch_prober.go:28] interesting pod/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 29 21:27:02.710526 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:02.709996 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 29 21:27:03.710537 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:03.710495 2582 patch_prober.go:28] interesting pod/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 29 21:27:03.711041 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:03.710598 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 29 21:27:04.713535 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:04.713507 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:27:04.713896 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:04.713773 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:27:04.714542 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:04.714526 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:27:09.687895 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.687862 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2"] Jul 29 21:27:09.690493 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.690475 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.693618 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.693587 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-60a3ea-openshift-default-dockercfg-kvgrp\"" Jul 29 21:27:09.715596 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.715566 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2"] Jul 29 21:27:09.818643 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.818609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-workload-socket\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.818643 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.818646 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-workload-certs\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.818927 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.818672 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-credential-socket\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.818927 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.818765 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-podinfo\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.818927 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.818861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-data\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.818927 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.818896 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-token\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.819153 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.818928 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-envoy\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.819153 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.818963 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istiod-ca-cert\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.819153 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.819045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kv2p\" (UniqueName: \"kubernetes.io/projected/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-kube-api-access-4kv2p\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.920512 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.920470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-credential-socket\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.920664 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.920539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-podinfo\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.920664 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.920604 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-data\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.920664 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.920624 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-token\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.920664 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.920648 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-envoy\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.920883 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.920672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istiod-ca-cert\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.920883 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.920701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kv2p\" (UniqueName: \"kubernetes.io/projected/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-kube-api-access-4kv2p\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.920883 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.920778 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-workload-socket\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.920883 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.920805 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-workload-certs\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.921162 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.920956 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-credential-socket\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.921302 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.921276 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-data\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.921437 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.921310 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-workload-certs\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.921437 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.921335 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-workload-socket\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.921544 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.921526 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istiod-ca-cert\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.923269 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.923245 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-podinfo\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.923399 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.923382 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-envoy\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.933792 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.933775 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-token\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:09.933888 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:09.933820 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kv2p\" (UniqueName: \"kubernetes.io/projected/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-kube-api-access-4kv2p\") pod \"e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:10.002041 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:10.001992 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:10.142490 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:10.142465 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2"] Jul 29 21:27:10.144097 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:27:10.144061 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8b5efb_3ee8_43bb_bf6d_2d9c420539dd.slice/crio-80cf9d1e41b011f031180fb070b04e770e3d09220a72162f4751560f578741af WatchSource:0}: Error finding container 80cf9d1e41b011f031180fb070b04e770e3d09220a72162f4751560f578741af: Status 404 returned error can't find the container with id 80cf9d1e41b011f031180fb070b04e770e3d09220a72162f4751560f578741af Jul 29 21:27:10.146370 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:10.146336 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:27:10.146456 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:10.146398 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:27:10.146456 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:10.146426 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:27:10.176409 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:10.176379 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" event={"ID":"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd","Type":"ContainerStarted","Data":"80cf9d1e41b011f031180fb070b04e770e3d09220a72162f4751560f578741af"} Jul 29 21:27:11.183626 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:11.183590 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" event={"ID":"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd","Type":"ContainerStarted","Data":"cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb"} Jul 29 21:27:11.207932 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:11.207881 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" podStartSLOduration=2.2078661569999998 podStartE2EDuration="2.207866157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:27:11.205659357 +0000 UTC m=+2379.329249200" watchObservedRunningTime="2026-07-29 21:27:11.207866157 +0000 UTC m=+2379.331456003" Jul 29 21:27:12.002610 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:12.002571 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:12.007930 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:12.007901 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:12.188715 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:12.188686 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:12.189633 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:12.189612 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:15.001310 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.001274 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676"] Jul 29 21:27:15.004064 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.004017 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" Jul 29 21:27:15.006790 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.006766 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-60a3ea\"" Jul 29 21:27:15.006914 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.006788 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-60a3ea-dockercfg-99hz4\"" Jul 29 21:27:15.021081 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.021058 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676"] Jul 29 21:27:15.067099 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.067072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a104144-b8c1-42f2-b6f4-75097b5b6ef7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-60a3ea-6594465676-5l676\" (UID: \"6a104144-b8c1-42f2-b6f4-75097b5b6ef7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" Jul 29 21:27:15.067226 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.067104 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzq2k\" (UniqueName: \"kubernetes.io/projected/6a104144-b8c1-42f2-b6f4-75097b5b6ef7-kube-api-access-jzq2k\") pod \"payload-processing-e2e-inf-b-60a3ea-6594465676-5l676\" (UID: \"6a104144-b8c1-42f2-b6f4-75097b5b6ef7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" Jul 29 21:27:15.167869 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.167831 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a104144-b8c1-42f2-b6f4-75097b5b6ef7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-60a3ea-6594465676-5l676\" (UID: \"6a104144-b8c1-42f2-b6f4-75097b5b6ef7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" Jul 29 21:27:15.167869 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.167869 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzq2k\" (UniqueName: \"kubernetes.io/projected/6a104144-b8c1-42f2-b6f4-75097b5b6ef7-kube-api-access-jzq2k\") pod \"payload-processing-e2e-inf-b-60a3ea-6594465676-5l676\" (UID: \"6a104144-b8c1-42f2-b6f4-75097b5b6ef7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" Jul 29 21:27:15.168619 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.168601 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a104144-b8c1-42f2-b6f4-75097b5b6ef7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-60a3ea-6594465676-5l676\" (UID: \"6a104144-b8c1-42f2-b6f4-75097b5b6ef7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" Jul 29 21:27:15.176672 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.176646 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzq2k\" (UniqueName: \"kubernetes.io/projected/6a104144-b8c1-42f2-b6f4-75097b5b6ef7-kube-api-access-jzq2k\") pod \"payload-processing-e2e-inf-b-60a3ea-6594465676-5l676\" (UID: \"6a104144-b8c1-42f2-b6f4-75097b5b6ef7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" Jul 29 21:27:15.315104 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.315003 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" Jul 29 21:27:15.453621 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:15.453594 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676"] Jul 29 21:27:15.454801 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:27:15.454776 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a104144_b8c1_42f2_b6f4_75097b5b6ef7.slice/crio-d9072263a685d9a39a8abb27fe824a9aca1580a995333117e557efa78a92c988 WatchSource:0}: Error finding container d9072263a685d9a39a8abb27fe824a9aca1580a995333117e557efa78a92c988: Status 404 returned error can't find the container with id d9072263a685d9a39a8abb27fe824a9aca1580a995333117e557efa78a92c988 Jul 29 21:27:16.213799 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:16.213762 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" event={"ID":"6a104144-b8c1-42f2-b6f4-75097b5b6ef7","Type":"ContainerStarted","Data":"b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a"} Jul 29 21:27:16.214300 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:16.213829 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" Jul 29 21:27:16.214300 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:16.213849 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" event={"ID":"6a104144-b8c1-42f2-b6f4-75097b5b6ef7","Type":"ContainerStarted","Data":"d9072263a685d9a39a8abb27fe824a9aca1580a995333117e557efa78a92c988"} Jul 29 21:27:16.231953 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:16.231906 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" podStartSLOduration=2.231890217 podStartE2EDuration="2.231890217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:27:16.230771044 +0000 UTC m=+2384.354360899" watchObservedRunningTime="2026-07-29 21:27:16.231890217 +0000 UTC m=+2384.355480094" Jul 29 21:27:20.947844 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:20.947807 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:27:27.222362 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:27.222329 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" Jul 29 21:27:28.230211 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:28.230167 2582 patch_prober.go:28] interesting pod/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 21:27:28.230612 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:28.230251 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 21:27:29.284796 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:29.284764 2582 generic.go:356] "Generic (PLEG): container finished" podID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerID="cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb" exitCode=0 Jul 29 21:27:29.285227 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:29.284838 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" event={"ID":"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd","Type":"ContainerDied","Data":"cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb"} Jul 29 21:27:29.285342 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:29.285325 2582 scope.go:117] "RemoveContainer" containerID="cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb" Jul 29 21:27:29.286370 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:29.286339 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:27:29.286481 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:29.286425 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:27:29.286481 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:29.286469 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:27:30.302069 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:30.302003 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" event={"ID":"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd","Type":"ContainerStarted","Data":"b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e"} Jul 29 21:27:31.003364 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:31.003319 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:31.003607 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:31.003458 2582 patch_prober.go:28] interesting pod/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Jul 29 21:27:31.003607 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:31.003502 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Jul 29 21:27:32.002962 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:32.002918 2582 patch_prober.go:28] interesting pod/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Jul 29 21:27:32.003507 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:32.002992 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Jul 29 21:27:33.002626 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:33.002580 2582 patch_prober.go:28] interesting pod/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Jul 29 21:27:33.002893 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:33.002679 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Jul 29 21:27:34.010555 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:34.010510 2582 patch_prober.go:28] interesting pod/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Jul 29 21:27:34.011190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:34.011156 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Jul 29 21:27:35.006754 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:35.006714 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:35.006998 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:35.006978 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:35.007839 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:35.007821 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:27:53.864622 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:27:53.864535 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:28:05.459612 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:05.459573 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:28:16.061244 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:16.061205 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:28:59.083516 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:59.083484 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl"] Jul 29 21:28:59.086454 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:59.086437 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl" Jul 29 21:28:59.090517 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:59.090491 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:28:59.090668 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:59.090497 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\"" Jul 29 21:28:59.101590 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:59.101561 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl"] Jul 29 21:28:59.263436 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:59.263395 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33e4ef7-db05-4d94-8d6f-ea2a70d18e75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl\" (UID: \"f33e4ef7-db05-4d94-8d6f-ea2a70d18e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl" Jul 29 21:28:59.364795 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:59.364694 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33e4ef7-db05-4d94-8d6f-ea2a70d18e75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl\" (UID: \"f33e4ef7-db05-4d94-8d6f-ea2a70d18e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl" Jul 29 21:28:59.365400 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:59.365380 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33e4ef7-db05-4d94-8d6f-ea2a70d18e75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl\" (UID: \"f33e4ef7-db05-4d94-8d6f-ea2a70d18e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl" Jul 29 21:28:59.396889 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:59.396852 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl" Jul 29 21:28:59.548526 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:59.548490 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl"] Jul 29 21:28:59.550505 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:28:59.550480 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33e4ef7_db05_4d94_8d6f_ea2a70d18e75.slice/crio-9381952b1af730550e8fe10e3a8b0a4f1ebc6a59e975f4c76a043e89081c2595 WatchSource:0}: Error finding container 9381952b1af730550e8fe10e3a8b0a4f1ebc6a59e975f4c76a043e89081c2595: Status 404 returned error can't find the container with id 9381952b1af730550e8fe10e3a8b0a4f1ebc6a59e975f4c76a043e89081c2595 Jul 29 21:28:59.792837 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:28:59.792804 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl" event={"ID":"f33e4ef7-db05-4d94-8d6f-ea2a70d18e75","Type":"ContainerStarted","Data":"9381952b1af730550e8fe10e3a8b0a4f1ebc6a59e975f4c76a043e89081c2595"} Jul 29 21:29:00.800631 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:00.800595 2582 generic.go:356] "Generic (PLEG): container finished" podID="f33e4ef7-db05-4d94-8d6f-ea2a70d18e75" containerID="c97e08b5c2713aa10741d33bd13877720be8957edb671bd6df8ee4a4c5b71861" exitCode=0 Jul 29 21:29:00.801116 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:00.800639 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl" event={"ID":"f33e4ef7-db05-4d94-8d6f-ea2a70d18e75","Type":"ContainerDied","Data":"c97e08b5c2713aa10741d33bd13877720be8957edb671bd6df8ee4a4c5b71861"} Jul 29 21:29:01.950468 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:01.950442 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl" Jul 29 21:29:01.986373 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:01.986334 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33e4ef7-db05-4d94-8d6f-ea2a70d18e75-maas-api-service-ca\") pod \"f33e4ef7-db05-4d94-8d6f-ea2a70d18e75\" (UID: \"f33e4ef7-db05-4d94-8d6f-ea2a70d18e75\") " Jul 29 21:29:01.986704 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:01.986683 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33e4ef7-db05-4d94-8d6f-ea2a70d18e75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f33e4ef7-db05-4d94-8d6f-ea2a70d18e75" (UID: "f33e4ef7-db05-4d94-8d6f-ea2a70d18e75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:29:02.087469 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:02.087375 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33e4ef7-db05-4d94-8d6f-ea2a70d18e75-maas-api-service-ca\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:02.662527 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:02.662486 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:29:02.812907 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:02.812818 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl" Jul 29 21:29:02.813095 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:02.812826 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl" event={"ID":"f33e4ef7-db05-4d94-8d6f-ea2a70d18e75","Type":"ContainerDied","Data":"9381952b1af730550e8fe10e3a8b0a4f1ebc6a59e975f4c76a043e89081c2595"} Jul 29 21:29:02.813095 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:02.812945 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9381952b1af730550e8fe10e3a8b0a4f1ebc6a59e975f4c76a043e89081c2595" Jul 29 21:29:09.626514 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.626478 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9"] Jul 29 21:29:09.627043 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.626767 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" podUID="fa858ba9-4961-4a69-9eea-9135eb4ade30" containerName="payload-processing" containerID="cri-o://4dc3fcd22f594679e46df90f56f409453105cde1f30b6d4e9a4361a1afed2004" gracePeriod=30 Jul 29 21:29:09.637445 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.637414 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b"] Jul 29 21:29:09.637678 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.637644 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" podUID="b1cb5391-1db3-4a74-b593-720b28211502" containerName="payload-pre-processing" containerID="cri-o://331bfbc5ab15ce19e04cf940f025e3139f6fbe403b243a343fb1f738001dfd7d" gracePeriod=30 Jul 29 21:29:09.786485 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.786421 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl"] Jul 29 21:29:09.789873 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.789816 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d12bd-jnfvl"] Jul 29 21:29:09.853035 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.852895 2582 generic.go:356] "Generic (PLEG): container finished" podID="fa858ba9-4961-4a69-9eea-9135eb4ade30" containerID="4dc3fcd22f594679e46df90f56f409453105cde1f30b6d4e9a4361a1afed2004" exitCode=0 Jul 29 21:29:09.853035 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.852977 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" event={"ID":"fa858ba9-4961-4a69-9eea-9135eb4ade30","Type":"ContainerDied","Data":"4dc3fcd22f594679e46df90f56f409453105cde1f30b6d4e9a4361a1afed2004"} Jul 29 21:29:09.855578 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.855013 2582 generic.go:356] "Generic (PLEG): container finished" podID="b1cb5391-1db3-4a74-b593-720b28211502" containerID="331bfbc5ab15ce19e04cf940f025e3139f6fbe403b243a343fb1f738001dfd7d" exitCode=0 Jul 29 21:29:09.855578 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.855061 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" event={"ID":"b1cb5391-1db3-4a74-b593-720b28211502","Type":"ContainerDied","Data":"331bfbc5ab15ce19e04cf940f025e3139f6fbe403b243a343fb1f738001dfd7d"} Jul 29 21:29:09.920763 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.920738 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" Jul 29 21:29:09.942631 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.942601 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" Jul 29 21:29:09.961707 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.961678 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txr6r\" (UniqueName: \"kubernetes.io/projected/fa858ba9-4961-4a69-9eea-9135eb4ade30-kube-api-access-txr6r\") pod \"fa858ba9-4961-4a69-9eea-9135eb4ade30\" (UID: \"fa858ba9-4961-4a69-9eea-9135eb4ade30\") " Jul 29 21:29:09.961891 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.961749 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa858ba9-4961-4a69-9eea-9135eb4ade30-plugins-config-volume\") pod \"fa858ba9-4961-4a69-9eea-9135eb4ade30\" (UID: \"fa858ba9-4961-4a69-9eea-9135eb4ade30\") " Jul 29 21:29:09.962222 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.962188 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa858ba9-4961-4a69-9eea-9135eb4ade30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa858ba9-4961-4a69-9eea-9135eb4ade30" (UID: "fa858ba9-4961-4a69-9eea-9135eb4ade30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:29:09.964302 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:09.964275 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa858ba9-4961-4a69-9eea-9135eb4ade30-kube-api-access-txr6r" (OuterVolumeSpecName: "kube-api-access-txr6r") pod "fa858ba9-4961-4a69-9eea-9135eb4ade30" (UID: "fa858ba9-4961-4a69-9eea-9135eb4ade30"). InnerVolumeSpecName "kube-api-access-txr6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:29:10.063208 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.063165 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb5391-1db3-4a74-b593-720b28211502-plugins-config-volume\") pod \"b1cb5391-1db3-4a74-b593-720b28211502\" (UID: \"b1cb5391-1db3-4a74-b593-720b28211502\") " Jul 29 21:29:10.063422 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.063232 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqb46\" (UniqueName: \"kubernetes.io/projected/b1cb5391-1db3-4a74-b593-720b28211502-kube-api-access-kqb46\") pod \"b1cb5391-1db3-4a74-b593-720b28211502\" (UID: \"b1cb5391-1db3-4a74-b593-720b28211502\") " Jul 29 21:29:10.063492 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.063469 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa858ba9-4961-4a69-9eea-9135eb4ade30-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:10.063492 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.063482 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txr6r\" (UniqueName: \"kubernetes.io/projected/fa858ba9-4961-4a69-9eea-9135eb4ade30-kube-api-access-txr6r\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:10.063678 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.063650 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cb5391-1db3-4a74-b593-720b28211502-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1cb5391-1db3-4a74-b593-720b28211502" (UID: "b1cb5391-1db3-4a74-b593-720b28211502"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:29:10.065530 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.065506 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cb5391-1db3-4a74-b593-720b28211502-kube-api-access-kqb46" (OuterVolumeSpecName: "kube-api-access-kqb46") pod "b1cb5391-1db3-4a74-b593-720b28211502" (UID: "b1cb5391-1db3-4a74-b593-720b28211502"). InnerVolumeSpecName "kube-api-access-kqb46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:29:10.164173 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.164079 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb5391-1db3-4a74-b593-720b28211502-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:10.164173 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.164113 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqb46\" (UniqueName: \"kubernetes.io/projected/b1cb5391-1db3-4a74-b593-720b28211502-kube-api-access-kqb46\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:10.494489 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.494444 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f33e4ef7-db05-4d94-8d6f-ea2a70d18e75" path="/var/lib/kubelet/pods/f33e4ef7-db05-4d94-8d6f-ea2a70d18e75/volumes" Jul 29 21:29:10.862150 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.862062 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" Jul 29 21:29:10.862622 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.862063 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b" event={"ID":"b1cb5391-1db3-4a74-b593-720b28211502","Type":"ContainerDied","Data":"6c23b129d497c2fb4c745717f798c240d435e79613f1468e4797dc25245c5c7c"} Jul 29 21:29:10.862622 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.862208 2582 scope.go:117] "RemoveContainer" containerID="331bfbc5ab15ce19e04cf940f025e3139f6fbe403b243a343fb1f738001dfd7d" Jul 29 21:29:10.863643 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.863535 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" Jul 29 21:29:10.863643 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.863539 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9" event={"ID":"fa858ba9-4961-4a69-9eea-9135eb4ade30","Type":"ContainerDied","Data":"bdca623221c9032409d3ff23d9f63c68e29b1fb051a00c5d3fb401152e578b20"} Jul 29 21:29:10.871785 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.871765 2582 scope.go:117] "RemoveContainer" containerID="4dc3fcd22f594679e46df90f56f409453105cde1f30b6d4e9a4361a1afed2004" Jul 29 21:29:10.882724 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.882699 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b"] Jul 29 21:29:10.890065 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.890018 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d12bd-5c6687b888-b6q4b"] Jul 29 21:29:10.901188 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.901162 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9"] Jul 29 21:29:10.905712 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:10.905686 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d12bd-6475c57fc6-w2bw9"] Jul 29 21:29:12.493843 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:12.493805 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1cb5391-1db3-4a74-b593-720b28211502" path="/var/lib/kubelet/pods/b1cb5391-1db3-4a74-b593-720b28211502/volumes" Jul 29 21:29:12.494359 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:12.494175 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa858ba9-4961-4a69-9eea-9135eb4ade30" path="/var/lib/kubelet/pods/fa858ba9-4961-4a69-9eea-9135eb4ade30/volumes" Jul 29 21:29:16.250802 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:16.250764 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv"] Jul 29 21:29:16.251354 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:16.251006 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" containerID="cri-o://e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2" gracePeriod=30 Jul 29 21:29:19.252133 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:19.252096 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:29:19.716756 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:19.716664 2582 patch_prober.go:28] interesting pod/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 21:29:19.716756 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:19.716724 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 21:29:21.516877 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.516807 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:29:21.586508 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.586466 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9qv2\" (UniqueName: \"kubernetes.io/projected/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-kube-api-access-b9qv2\") pod \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " Jul 29 21:29:21.586730 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.586529 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istiod-ca-cert\") pod \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " Jul 29 21:29:21.586730 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.586572 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-data\") pod \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " Jul 29 21:29:21.586730 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.586613 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-credential-socket\") pod \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " Jul 29 21:29:21.586944 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.586747 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-workload-certs\") pod \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " Jul 29 21:29:21.586944 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.586792 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-envoy\") pod \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " Jul 29 21:29:21.586944 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.586851 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-podinfo\") pod \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " Jul 29 21:29:21.586944 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.586909 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" (UID: "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:29:21.587248 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.587004 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" (UID: "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:29:21.587248 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.587017 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-workload-socket\") pod \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " Jul 29 21:29:21.587248 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.587145 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-token\") pod \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\" (UID: \"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c\") " Jul 29 21:29:21.587701 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.587674 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" (UID: "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:29:21.587701 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.587687 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" (UID: "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:29:21.587865 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.587660 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:21.587865 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.587676 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" (UID: "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:29:21.587865 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.587732 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:21.589156 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.589127 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" (UID: "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:29:21.589277 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.589216 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-kube-api-access-b9qv2" (OuterVolumeSpecName: "kube-api-access-b9qv2") pod "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" (UID: "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c"). InnerVolumeSpecName "kube-api-access-b9qv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:29:21.589619 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.589600 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" (UID: "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:29:21.589721 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.589707 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" (UID: "4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:29:21.688586 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.688548 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:21.688586 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.688579 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:21.688586 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.688589 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9qv2\" (UniqueName: \"kubernetes.io/projected/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-kube-api-access-b9qv2\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:21.688586 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.688599 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:21.688876 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.688608 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:21.688876 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.688615 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:21.688876 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.688622 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:21.925584 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.925488 2582 generic.go:356] "Generic (PLEG): container finished" podID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerID="e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2" exitCode=0 Jul 29 21:29:21.925584 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.925548 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" Jul 29 21:29:21.925584 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.925566 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" event={"ID":"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c","Type":"ContainerDied","Data":"e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2"} Jul 29 21:29:21.925850 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.925657 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv" event={"ID":"4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c","Type":"ContainerDied","Data":"cb3121b0f19b0e6a60e8a513e8c65b99235a5006b4d8a990d17bacacc1080a40"} Jul 29 21:29:21.925850 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.925690 2582 scope.go:117] "RemoveContainer" containerID="e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2" Jul 29 21:29:21.935870 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.935853 2582 scope.go:117] "RemoveContainer" containerID="4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e" Jul 29 21:29:21.945087 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.945066 2582 scope.go:117] "RemoveContainer" containerID="e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2" Jul 29 21:29:21.945372 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:29:21.945349 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2\": container with ID starting with e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2 not found: ID does not exist" containerID="e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2" Jul 29 21:29:21.945462 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.945386 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2"} err="failed to get container status \"e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2\": rpc error: code = NotFound desc = could not find container \"e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2\": container with ID starting with e6536d49475cc505c60bcbe6cc6454a4cebca55593eebe893d1c014647d046e2 not found: ID does not exist" Jul 29 21:29:21.945462 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.945412 2582 scope.go:117] "RemoveContainer" containerID="4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e" Jul 29 21:29:21.945679 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:29:21.945661 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e\": container with ID starting with 4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e not found: ID does not exist" containerID="4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e" Jul 29 21:29:21.945727 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.945687 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e"} err="failed to get container status \"4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e\": rpc error: code = NotFound desc = could not find container \"4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e\": container with ID starting with 4c489092aee15a6471e0a12ee643a8835706ed54fdd8a294b814ce669510bc4e not found: ID does not exist" Jul 29 21:29:21.949749 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.949727 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv"] Jul 29 21:29:21.953888 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:21.953865 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4d12bd-openshift-default-6db567649b-scszv"] Jul 29 21:29:22.493404 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:22.493360 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" path="/var/lib/kubelet/pods/4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c/volumes" Jul 29 21:29:27.179541 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.179452 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676"] Jul 29 21:29:27.180065 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.179733 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" podUID="6a104144-b8c1-42f2-b6f4-75097b5b6ef7" containerName="payload-processing" containerID="cri-o://b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a" gracePeriod=30 Jul 29 21:29:27.220774 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.220729 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" podUID="6a104144-b8c1-42f2-b6f4-75097b5b6ef7" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.76:9004: connect: connection refused" Jul 29 21:29:27.449216 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.449181 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" Jul 29 21:29:27.544872 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.544841 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a104144-b8c1-42f2-b6f4-75097b5b6ef7-plugins-config-volume\") pod \"6a104144-b8c1-42f2-b6f4-75097b5b6ef7\" (UID: \"6a104144-b8c1-42f2-b6f4-75097b5b6ef7\") " Jul 29 21:29:27.545093 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.544979 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzq2k\" (UniqueName: \"kubernetes.io/projected/6a104144-b8c1-42f2-b6f4-75097b5b6ef7-kube-api-access-jzq2k\") pod \"6a104144-b8c1-42f2-b6f4-75097b5b6ef7\" (UID: \"6a104144-b8c1-42f2-b6f4-75097b5b6ef7\") " Jul 29 21:29:27.545258 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.545233 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a104144-b8c1-42f2-b6f4-75097b5b6ef7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a104144-b8c1-42f2-b6f4-75097b5b6ef7" (UID: "6a104144-b8c1-42f2-b6f4-75097b5b6ef7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:29:27.545337 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.545316 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a104144-b8c1-42f2-b6f4-75097b5b6ef7-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:27.547181 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.547158 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a104144-b8c1-42f2-b6f4-75097b5b6ef7-kube-api-access-jzq2k" (OuterVolumeSpecName: "kube-api-access-jzq2k") pod "6a104144-b8c1-42f2-b6f4-75097b5b6ef7" (UID: "6a104144-b8c1-42f2-b6f4-75097b5b6ef7"). InnerVolumeSpecName "kube-api-access-jzq2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:29:27.646797 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.646762 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzq2k\" (UniqueName: \"kubernetes.io/projected/6a104144-b8c1-42f2-b6f4-75097b5b6ef7-kube-api-access-jzq2k\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:27.959101 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.959068 2582 generic.go:356] "Generic (PLEG): container finished" podID="6a104144-b8c1-42f2-b6f4-75097b5b6ef7" containerID="b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a" exitCode=0 Jul 29 21:29:27.959283 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.959112 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" Jul 29 21:29:27.959283 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.959156 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" event={"ID":"6a104144-b8c1-42f2-b6f4-75097b5b6ef7","Type":"ContainerDied","Data":"b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a"} Jul 29 21:29:27.959283 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.959216 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676" event={"ID":"6a104144-b8c1-42f2-b6f4-75097b5b6ef7","Type":"ContainerDied","Data":"d9072263a685d9a39a8abb27fe824a9aca1580a995333117e557efa78a92c988"} Jul 29 21:29:27.959283 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.959239 2582 scope.go:117] "RemoveContainer" containerID="b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a" Jul 29 21:29:27.968655 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.968637 2582 scope.go:117] "RemoveContainer" containerID="b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a" Jul 29 21:29:27.968949 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:29:27.968930 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a\": container with ID starting with b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a not found: ID does not exist" containerID="b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a" Jul 29 21:29:27.969041 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.968970 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a"} err="failed to get container status \"b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a\": rpc error: code = NotFound desc = could not find container \"b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a\": container with ID starting with b0ad883e66066db7b6756bf7cfceb97be29aee7400ad537c1d90f60632cacb5a not found: ID does not exist" Jul 29 21:29:27.980718 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.980691 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676"] Jul 29 21:29:27.985039 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:27.985003 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-60a3ea-6594465676-5l676"] Jul 29 21:29:28.493445 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:28.493410 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a104144-b8c1-42f2-b6f4-75097b5b6ef7" path="/var/lib/kubelet/pods/6a104144-b8c1-42f2-b6f4-75097b5b6ef7/volumes" Jul 29 21:29:30.857794 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:30.857756 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:29:32.731041 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:32.730997 2582 scope.go:117] "RemoveContainer" containerID="3b93af0065a191607b03a14f0a3766eea9b53fe555863a7563a8824d68c170ad" Jul 29 21:29:34.132418 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:34.132383 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2"] Jul 29 21:29:34.132820 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:34.132620 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" containerID="cri-o://b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e" gracePeriod=30 Jul 29 21:29:35.006447 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.006415 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz"] Jul 29 21:29:35.006869 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.006846 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" Jul 29 21:29:35.006869 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.006870 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" Jul 29 21:29:35.007118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.006889 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a104144-b8c1-42f2-b6f4-75097b5b6ef7" containerName="payload-processing" Jul 29 21:29:35.007118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.006899 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a104144-b8c1-42f2-b6f4-75097b5b6ef7" containerName="payload-processing" Jul 29 21:29:35.007118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.006915 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f33e4ef7-db05-4d94-8d6f-ea2a70d18e75" containerName="revoke-keys" Jul 29 21:29:35.007118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.006924 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f33e4ef7-db05-4d94-8d6f-ea2a70d18e75" containerName="revoke-keys" Jul 29 21:29:35.007118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.006938 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" Jul 29 21:29:35.007118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.006946 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" Jul 29 21:29:35.007118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.006960 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa858ba9-4961-4a69-9eea-9135eb4ade30" containerName="payload-processing" Jul 29 21:29:35.007118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.006969 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa858ba9-4961-4a69-9eea-9135eb4ade30" containerName="payload-processing" Jul 29 21:29:35.007118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.007010 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1cb5391-1db3-4a74-b593-720b28211502" containerName="payload-pre-processing" Jul 29 21:29:35.007118 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.007037 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1cb5391-1db3-4a74-b593-720b28211502" containerName="payload-pre-processing" Jul 29 21:29:35.007592 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.007144 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" Jul 29 21:29:35.007592 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.007157 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa858ba9-4961-4a69-9eea-9135eb4ade30" containerName="payload-processing" Jul 29 21:29:35.007592 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.007202 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1cb5391-1db3-4a74-b593-720b28211502" containerName="payload-pre-processing" Jul 29 21:29:35.007592 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.007210 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a13bfcf-2b73-40bb-8de5-7eb2cd00a38c" containerName="istio-proxy" Jul 29 21:29:35.007592 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.007217 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f33e4ef7-db05-4d94-8d6f-ea2a70d18e75" containerName="revoke-keys" Jul 29 21:29:35.007592 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.007226 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a104144-b8c1-42f2-b6f4-75097b5b6ef7" containerName="payload-processing" Jul 29 21:29:35.009679 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.009653 2582 patch_prober.go:28] interesting pod/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 21:29:35.009797 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.009739 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 21:29:35.011429 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.011412 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.013935 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.013916 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a7e568-openshift-default-dockercfg-lswwc\"" Jul 29 21:29:35.016094 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.016067 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-workload-certs\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.016171 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.016120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-credential-socket\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.016171 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.016147 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-envoy\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.016266 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.016244 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istiod-ca-cert\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.016309 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.016297 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-token\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.016342 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.016335 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-data\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.016380 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.016361 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-podinfo\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.016419 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.016402 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxh5b\" (UniqueName: \"kubernetes.io/projected/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-kube-api-access-qxh5b\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.016451 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.016428 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-workload-socket\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.022537 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.022511 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz"] Jul 29 21:29:35.117688 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.117649 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-workload-certs\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.117877 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.117699 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-credential-socket\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.117877 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.117721 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-envoy\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.117877 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.117765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istiod-ca-cert\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.117877 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.117802 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-token\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.117877 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.117850 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-data\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.118198 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.117886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-podinfo\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.118198 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.118143 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-workload-certs\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.118198 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.118166 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-credential-socket\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.118363 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.118205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxh5b\" (UniqueName: \"kubernetes.io/projected/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-kube-api-access-qxh5b\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.118363 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.118235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-workload-socket\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.118482 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.118422 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-data\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.118545 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.118533 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-workload-socket\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.118633 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.118601 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istiod-ca-cert\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.121058 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.121013 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-envoy\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.121307 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.121288 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-podinfo\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.126764 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.126744 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-token\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.127084 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.127016 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxh5b\" (UniqueName: \"kubernetes.io/projected/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-kube-api-access-qxh5b\") pod \"e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.323917 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.323813 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:35.472677 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.472651 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz"] Jul 29 21:29:35.474786 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:29:35.474761 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee86d0f_e718_4b8a_84fe_3ceb09db5a65.slice/crio-c190a9a0bc5f3c619c141e3e844c2cb19bd339f26bcd2ff0ef96ca050108169b WatchSource:0}: Error finding container c190a9a0bc5f3c619c141e3e844c2cb19bd339f26bcd2ff0ef96ca050108169b: Status 404 returned error can't find the container with id c190a9a0bc5f3c619c141e3e844c2cb19bd339f26bcd2ff0ef96ca050108169b Jul 29 21:29:35.477489 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.477453 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:29:35.477591 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.477524 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:29:35.477591 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:35.477552 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:29:36.016271 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:36.016231 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" event={"ID":"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65","Type":"ContainerStarted","Data":"0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12"} Jul 29 21:29:36.016487 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:36.016290 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" event={"ID":"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65","Type":"ContainerStarted","Data":"c190a9a0bc5f3c619c141e3e844c2cb19bd339f26bcd2ff0ef96ca050108169b"} Jul 29 21:29:36.041498 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:36.041423 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" podStartSLOduration=2.041398641 podStartE2EDuration="2.041398641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:29:36.040796976 +0000 UTC m=+2524.164386819" watchObservedRunningTime="2026-07-29 21:29:36.041398641 +0000 UTC m=+2524.164988487" Jul 29 21:29:36.324056 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:36.323927 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:36.329368 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:36.329339 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:37.021333 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:37.021302 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:37.022271 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:37.022252 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:39.382732 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.382707 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:29:39.459150 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459049 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-data\") pod \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " Jul 29 21:29:39.459150 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459106 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istiod-ca-cert\") pod \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " Jul 29 21:29:39.459150 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459150 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-workload-certs\") pod \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " Jul 29 21:29:39.459446 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459167 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-podinfo\") pod \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " Jul 29 21:29:39.459446 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459192 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-token\") pod \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " Jul 29 21:29:39.459446 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459230 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-envoy\") pod \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " Jul 29 21:29:39.459446 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459287 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-credential-socket\") pod \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " Jul 29 21:29:39.459446 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459336 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-workload-socket\") pod \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " Jul 29 21:29:39.459446 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459366 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kv2p\" (UniqueName: \"kubernetes.io/projected/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-kube-api-access-4kv2p\") pod \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\" (UID: \"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd\") " Jul 29 21:29:39.459753 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459466 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" (UID: "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:29:39.459753 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459552 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" (UID: "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:29:39.459753 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459705 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:39.459753 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459725 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:39.459941 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459876 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" (UID: "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:29:39.459941 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459891 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" (UID: "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:29:39.459941 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.459905 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" (UID: "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:29:39.461834 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.461797 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" (UID: "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:29:39.461977 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.461853 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" (UID: "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:29:39.462299 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.462057 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" (UID: "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:29:39.462387 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.462359 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-kube-api-access-4kv2p" (OuterVolumeSpecName: "kube-api-access-4kv2p") pod "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" (UID: "9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd"). InnerVolumeSpecName "kube-api-access-4kv2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:29:39.560667 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.560625 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:39.560667 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.560659 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:39.560667 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.560673 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:39.560904 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.560684 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:39.560904 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.560694 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:39.560904 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.560705 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kv2p\" (UniqueName: \"kubernetes.io/projected/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-kube-api-access-4kv2p\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:39.560904 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.560715 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:29:39.796217 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.796176 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv"] Jul 29 21:29:39.796728 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.796707 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" Jul 29 21:29:39.796728 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.796733 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" Jul 29 21:29:39.796876 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.796829 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" Jul 29 21:29:39.799147 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.799129 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" Jul 29 21:29:39.805374 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.805341 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a7e568-dockercfg-xljt2\"" Jul 29 21:29:39.805374 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.805358 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a7e568\"" Jul 29 21:29:39.811584 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.811558 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv"] Jul 29 21:29:39.853889 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.853853 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x"] Jul 29 21:29:39.854274 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.854259 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" Jul 29 21:29:39.854324 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.854276 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" Jul 29 21:29:39.854364 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.854344 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerName="istio-proxy" Jul 29 21:29:39.856670 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.856639 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" Jul 29 21:29:39.863980 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.863195 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4085153f-1520-4d91-95b5-9bbbc33b0902-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv\" (UID: \"4085153f-1520-4d91-95b5-9bbbc33b0902\") " pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" Jul 29 21:29:39.863980 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.863347 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84stn\" (UniqueName: \"kubernetes.io/projected/4085153f-1520-4d91-95b5-9bbbc33b0902-kube-api-access-84stn\") pod \"payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv\" (UID: \"4085153f-1520-4d91-95b5-9bbbc33b0902\") " pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" Jul 29 21:29:39.866398 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.866365 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x"] Jul 29 21:29:39.964659 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.964626 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmkvc\" (UniqueName: \"kubernetes.io/projected/93a432cd-51be-4524-a0c6-e3425fbd3219-kube-api-access-xmkvc\") pod \"payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x\" (UID: \"93a432cd-51be-4524-a0c6-e3425fbd3219\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" Jul 29 21:29:39.964878 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.964716 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84stn\" (UniqueName: \"kubernetes.io/projected/4085153f-1520-4d91-95b5-9bbbc33b0902-kube-api-access-84stn\") pod \"payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv\" (UID: \"4085153f-1520-4d91-95b5-9bbbc33b0902\") " pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" Jul 29 21:29:39.964878 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.964767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4085153f-1520-4d91-95b5-9bbbc33b0902-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv\" (UID: \"4085153f-1520-4d91-95b5-9bbbc33b0902\") " pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" Jul 29 21:29:39.964878 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.964813 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a432cd-51be-4524-a0c6-e3425fbd3219-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x\" (UID: \"93a432cd-51be-4524-a0c6-e3425fbd3219\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" Jul 29 21:29:39.965641 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.965615 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4085153f-1520-4d91-95b5-9bbbc33b0902-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv\" (UID: \"4085153f-1520-4d91-95b5-9bbbc33b0902\") " pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" Jul 29 21:29:39.974699 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:39.974664 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84stn\" (UniqueName: \"kubernetes.io/projected/4085153f-1520-4d91-95b5-9bbbc33b0902-kube-api-access-84stn\") pod \"payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv\" (UID: \"4085153f-1520-4d91-95b5-9bbbc33b0902\") " pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" Jul 29 21:29:40.039795 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.039765 2582 generic.go:356] "Generic (PLEG): container finished" podID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" containerID="b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e" exitCode=0 Jul 29 21:29:40.039795 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.039802 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" event={"ID":"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd","Type":"ContainerDied","Data":"b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e"} Jul 29 21:29:40.040060 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.039819 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" Jul 29 21:29:40.040060 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.039843 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2" event={"ID":"9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd","Type":"ContainerDied","Data":"80cf9d1e41b011f031180fb070b04e770e3d09220a72162f4751560f578741af"} Jul 29 21:29:40.040060 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.039867 2582 scope.go:117] "RemoveContainer" containerID="b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e" Jul 29 21:29:40.049328 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.049308 2582 scope.go:117] "RemoveContainer" containerID="cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb" Jul 29 21:29:40.057934 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.057914 2582 scope.go:117] "RemoveContainer" containerID="b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e" Jul 29 21:29:40.058217 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:29:40.058198 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e\": container with ID starting with b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e not found: ID does not exist" containerID="b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e" Jul 29 21:29:40.058266 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.058228 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e"} err="failed to get container status \"b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e\": rpc error: code = NotFound desc = could not find container \"b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e\": container with ID starting with b7ac5b9bec445dbb5ec5aee4d30c674da1e4fbfa09d2b8a02c75f8349e7aaf3e not found: ID does not exist" Jul 29 21:29:40.058266 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.058248 2582 scope.go:117] "RemoveContainer" containerID="cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb" Jul 29 21:29:40.058503 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:29:40.058483 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb\": container with ID starting with cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb not found: ID does not exist" containerID="cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb" Jul 29 21:29:40.058543 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.058511 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb"} err="failed to get container status \"cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb\": rpc error: code = NotFound desc = could not find container \"cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb\": container with ID starting with cab5338b9c3736223576550b44582b148ca374b8f3cf17f394897b20ecdf79eb not found: ID does not exist" Jul 29 21:29:40.062546 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.062522 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2"] Jul 29 21:29:40.066009 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.065985 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmkvc\" (UniqueName: \"kubernetes.io/projected/93a432cd-51be-4524-a0c6-e3425fbd3219-kube-api-access-xmkvc\") pod \"payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x\" (UID: \"93a432cd-51be-4524-a0c6-e3425fbd3219\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" Jul 29 21:29:40.066164 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.066120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a432cd-51be-4524-a0c6-e3425fbd3219-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x\" (UID: \"93a432cd-51be-4524-a0c6-e3425fbd3219\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" Jul 29 21:29:40.066738 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.066719 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a432cd-51be-4524-a0c6-e3425fbd3219-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x\" (UID: \"93a432cd-51be-4524-a0c6-e3425fbd3219\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" Jul 29 21:29:40.066947 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.066930 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-60a3ea-openshift-default-7646f5b8-x6kv2"] Jul 29 21:29:40.075041 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.075003 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmkvc\" (UniqueName: \"kubernetes.io/projected/93a432cd-51be-4524-a0c6-e3425fbd3219-kube-api-access-xmkvc\") pod \"payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x\" (UID: \"93a432cd-51be-4524-a0c6-e3425fbd3219\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" Jul 29 21:29:40.110516 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.110480 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" Jul 29 21:29:40.167399 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.167227 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" Jul 29 21:29:40.272865 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.272798 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv"] Jul 29 21:29:40.274010 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:29:40.273984 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4085153f_1520_4d91_95b5_9bbbc33b0902.slice/crio-0cff51f1b7d1f395ab8bc3e74350009e52a641d439d7d2ead04eeb71272fa696 WatchSource:0}: Error finding container 0cff51f1b7d1f395ab8bc3e74350009e52a641d439d7d2ead04eeb71272fa696: Status 404 returned error can't find the container with id 0cff51f1b7d1f395ab8bc3e74350009e52a641d439d7d2ead04eeb71272fa696 Jul 29 21:29:40.348361 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.348270 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x"] Jul 29 21:29:40.352396 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:29:40.352365 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a432cd_51be_4524_a0c6_e3425fbd3219.slice/crio-665e358a8b2b9418bf69d75fc304028b851187743f29c747c21661156d5eda06 WatchSource:0}: Error finding container 665e358a8b2b9418bf69d75fc304028b851187743f29c747c21661156d5eda06: Status 404 returned error can't find the container with id 665e358a8b2b9418bf69d75fc304028b851187743f29c747c21661156d5eda06 Jul 29 21:29:40.495292 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:40.495244 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd" path="/var/lib/kubelet/pods/9f8b5efb-3ee8-43bb-bf6d-2d9c420539dd/volumes" Jul 29 21:29:41.047052 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:41.046981 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" event={"ID":"93a432cd-51be-4524-a0c6-e3425fbd3219","Type":"ContainerStarted","Data":"066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429"} Jul 29 21:29:41.047279 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:41.047076 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" event={"ID":"93a432cd-51be-4524-a0c6-e3425fbd3219","Type":"ContainerStarted","Data":"665e358a8b2b9418bf69d75fc304028b851187743f29c747c21661156d5eda06"} Jul 29 21:29:41.047279 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:41.047214 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" Jul 29 21:29:41.048623 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:41.048553 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" event={"ID":"4085153f-1520-4d91-95b5-9bbbc33b0902","Type":"ContainerStarted","Data":"d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588"} Jul 29 21:29:41.048715 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:41.048625 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" event={"ID":"4085153f-1520-4d91-95b5-9bbbc33b0902","Type":"ContainerStarted","Data":"0cff51f1b7d1f395ab8bc3e74350009e52a641d439d7d2ead04eeb71272fa696"} Jul 29 21:29:41.048715 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:41.048655 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" Jul 29 21:29:41.062812 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:41.062761 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" podStartSLOduration=2.062746801 podStartE2EDuration="2.062746801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:29:41.061474645 +0000 UTC m=+2529.185064487" watchObservedRunningTime="2026-07-29 21:29:41.062746801 +0000 UTC m=+2529.186336643" Jul 29 21:29:41.078217 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:41.078166 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" podStartSLOduration=2.078149038 podStartE2EDuration="2.078149038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:29:41.076243836 +0000 UTC m=+2529.199833679" watchObservedRunningTime="2026-07-29 21:29:41.078149038 +0000 UTC m=+2529.201738897" Jul 29 21:29:45.658063 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:45.658015 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:29:52.058275 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:52.058236 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" Jul 29 21:29:52.058817 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:52.058550 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" Jul 29 21:29:53.064246 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:53.064208 2582 patch_prober.go:28] interesting pod/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 21:29:53.064649 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:53.064290 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 21:29:54.125787 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:54.125751 2582 generic.go:356] "Generic (PLEG): container finished" podID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerID="0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12" exitCode=0 Jul 29 21:29:54.126340 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:54.125824 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" event={"ID":"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65","Type":"ContainerDied","Data":"0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12"} Jul 29 21:29:54.126340 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:54.126243 2582 scope.go:117] "RemoveContainer" containerID="0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12" Jul 29 21:29:54.127253 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:54.127224 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:29:54.127381 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:54.127312 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:29:54.127381 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:54.127363 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:29:55.132578 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:55.132542 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" event={"ID":"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65","Type":"ContainerStarted","Data":"35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e"} Jul 29 21:29:55.324835 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:55.324803 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:29:55.325090 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:55.325068 2582 patch_prober.go:28] interesting pod/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Jul 29 21:29:55.325147 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:55.325116 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Jul 29 21:29:56.324488 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:56.324446 2582 patch_prober.go:28] interesting pod/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Jul 29 21:29:56.325005 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:56.324521 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Jul 29 21:29:57.324675 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:57.324636 2582 patch_prober.go:28] interesting pod/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Jul 29 21:29:57.325114 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:57.324696 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Jul 29 21:29:58.324580 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:58.324541 2582 patch_prober.go:28] interesting pod/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Jul 29 21:29:58.324773 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:58.324615 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Jul 29 21:29:59.324937 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:59.324902 2582 patch_prober.go:28] interesting pod/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Jul 29 21:29:59.325347 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:29:59.324979 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Jul 29 21:30:00.142015 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.141982 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r"] Jul 29 21:30:00.144964 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.144941 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qq5qd"] Jul 29 21:30:00.145102 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.145084 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r" Jul 29 21:30:00.147577 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.147557 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qq5qd" Jul 29 21:30:00.148199 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.148182 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\"" Jul 29 21:30:00.154580 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.154556 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r"] Jul 29 21:30:00.157655 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.157470 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r" Jul 29 21:30:00.161177 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.161111 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qq5qd"] Jul 29 21:30:00.168684 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.168661 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qq5qd" Jul 29 21:30:00.319686 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.319608 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r"] Jul 29 21:30:00.322202 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:30:00.322161 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03589c3_e519_49dd_9fa6_5abd81122038.slice/crio-65b3cf2c702c08c34f2af7b006060816cabed14b55b939083fcdabc771736f70 WatchSource:0}: Error finding container 65b3cf2c702c08c34f2af7b006060816cabed14b55b939083fcdabc771736f70: Status 404 returned error can't find the container with id 65b3cf2c702c08c34f2af7b006060816cabed14b55b939083fcdabc771736f70 Jul 29 21:30:00.328376 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.328355 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:30:00.329240 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.329099 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:30:00.330132 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.330116 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:30:00.344416 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:00.344394 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qq5qd"] Jul 29 21:30:00.346081 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:30:00.346018 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753d0899_a6ac_4059_befa_507d10de4609.slice/crio-c2ccec23eaa659b90d7aa429359eb4c54afa5edd27077c8fbd7e311f9a92aa6f WatchSource:0}: Error finding container c2ccec23eaa659b90d7aa429359eb4c54afa5edd27077c8fbd7e311f9a92aa6f: Status 404 returned error can't find the container with id c2ccec23eaa659b90d7aa429359eb4c54afa5edd27077c8fbd7e311f9a92aa6f Jul 29 21:30:01.166148 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:01.166107 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qq5qd" event={"ID":"753d0899-a6ac-4059-befa-507d10de4609","Type":"ContainerStarted","Data":"3842474f02a8a82561e7698723330714a9319a612560d2565f42d217e4efef8b"} Jul 29 21:30:01.166404 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:01.166173 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qq5qd" event={"ID":"753d0899-a6ac-4059-befa-507d10de4609","Type":"ContainerStarted","Data":"c2ccec23eaa659b90d7aa429359eb4c54afa5edd27077c8fbd7e311f9a92aa6f"} Jul 29 21:30:01.167718 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:01.167696 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r" event={"ID":"e03589c3-e519-49dd-9fa6-5abd81122038","Type":"ContainerStarted","Data":"693bf7c95ac327f98a579fd5a09c9b8d19bd88fd94d3278b1e54e55ee29b2ecb"} Jul 29 21:30:01.167826 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:01.167735 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r" event={"ID":"e03589c3-e519-49dd-9fa6-5abd81122038","Type":"ContainerStarted","Data":"65b3cf2c702c08c34f2af7b006060816cabed14b55b939083fcdabc771736f70"} Jul 29 21:30:01.182471 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:01.182422 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qq5qd" podStartSLOduration=1.182402881 podStartE2EDuration="1.182402881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:30:01.18170792 +0000 UTC m=+2549.305297767" watchObservedRunningTime="2026-07-29 21:30:01.182402881 +0000 UTC m=+2549.305992724" Jul 29 21:30:01.197945 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:01.197901 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r" podStartSLOduration=1.197888292 podStartE2EDuration="1.197888292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:30:01.197082499 +0000 UTC m=+2549.320672353" watchObservedRunningTime="2026-07-29 21:30:01.197888292 +0000 UTC m=+2549.321478151" Jul 29 21:30:02.175197 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:02.175158 2582 generic.go:356] "Generic (PLEG): container finished" podID="753d0899-a6ac-4059-befa-507d10de4609" containerID="3842474f02a8a82561e7698723330714a9319a612560d2565f42d217e4efef8b" exitCode=0 Jul 29 21:30:02.175614 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:02.175215 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qq5qd" event={"ID":"753d0899-a6ac-4059-befa-507d10de4609","Type":"ContainerDied","Data":"3842474f02a8a82561e7698723330714a9319a612560d2565f42d217e4efef8b"} Jul 29 21:30:02.176756 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:02.176730 2582 generic.go:356] "Generic (PLEG): container finished" podID="e03589c3-e519-49dd-9fa6-5abd81122038" containerID="693bf7c95ac327f98a579fd5a09c9b8d19bd88fd94d3278b1e54e55ee29b2ecb" exitCode=0 Jul 29 21:30:02.176875 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:02.176785 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r" event={"ID":"e03589c3-e519-49dd-9fa6-5abd81122038","Type":"ContainerDied","Data":"693bf7c95ac327f98a579fd5a09c9b8d19bd88fd94d3278b1e54e55ee29b2ecb"} Jul 29 21:30:03.341900 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.341872 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qq5qd" Jul 29 21:30:03.345015 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.344993 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r" Jul 29 21:30:03.742313 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.742277 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c"] Jul 29 21:30:03.742665 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.742653 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="753d0899-a6ac-4059-befa-507d10de4609" containerName="cleanup" Jul 29 21:30:03.742709 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.742667 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="753d0899-a6ac-4059-befa-507d10de4609" containerName="cleanup" Jul 29 21:30:03.742741 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.742708 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e03589c3-e519-49dd-9fa6-5abd81122038" containerName="cleanup" Jul 29 21:30:03.742741 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.742714 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e03589c3-e519-49dd-9fa6-5abd81122038" containerName="cleanup" Jul 29 21:30:03.742806 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.742779 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e03589c3-e519-49dd-9fa6-5abd81122038" containerName="cleanup" Jul 29 21:30:03.742806 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.742786 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="753d0899-a6ac-4059-befa-507d10de4609" containerName="cleanup" Jul 29 21:30:03.745117 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.745100 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.749738 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.749715 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3b326f-openshift-default-dockercfg-sv4km\"" Jul 29 21:30:03.763853 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.763814 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c"] Jul 29 21:30:03.889953 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.889913 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-credential-socket\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.890163 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.889961 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99862056-cb0c-4c8b-a894-99947b14043e-istiod-ca-cert\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.890163 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.889993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-workload-socket\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.890163 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.890057 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-workload-certs\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.890163 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.890085 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-istio-data\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.890163 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.890109 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-istio-envoy\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.890163 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.890126 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99862056-cb0c-4c8b-a894-99947b14043e-istio-token\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.890366 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.890208 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99862056-cb0c-4c8b-a894-99947b14043e-istio-podinfo\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.890366 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.890236 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2k4p\" (UniqueName: \"kubernetes.io/projected/99862056-cb0c-4c8b-a894-99947b14043e-kube-api-access-t2k4p\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.991223 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.991184 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-istio-envoy\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.991223 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.991229 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99862056-cb0c-4c8b-a894-99947b14043e-istio-token\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.991445 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.991271 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99862056-cb0c-4c8b-a894-99947b14043e-istio-podinfo\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.991445 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.991291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2k4p\" (UniqueName: \"kubernetes.io/projected/99862056-cb0c-4c8b-a894-99947b14043e-kube-api-access-t2k4p\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.991445 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.991314 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-credential-socket\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.991445 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.991343 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99862056-cb0c-4c8b-a894-99947b14043e-istiod-ca-cert\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.991445 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.991389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-workload-socket\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.991445 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.991423 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-workload-certs\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.991763 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.991467 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-istio-data\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.991763 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.991739 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-credential-socket\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.991885 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.991787 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-istio-data\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.992189 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.992102 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-workload-certs\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.992189 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.992149 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-workload-socket\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.992386 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.992344 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99862056-cb0c-4c8b-a894-99947b14043e-istiod-ca-cert\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.993846 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.993820 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-istio-envoy\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:03.994352 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:03.994333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99862056-cb0c-4c8b-a894-99947b14043e-istio-podinfo\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:04.000100 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.000073 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99862056-cb0c-4c8b-a894-99947b14043e-istio-token\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:04.000357 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.000334 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2k4p\" (UniqueName: \"kubernetes.io/projected/99862056-cb0c-4c8b-a894-99947b14043e-kube-api-access-t2k4p\") pod \"e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:04.055248 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.055212 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:04.189992 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.189964 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qq5qd" Jul 29 21:30:04.190203 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.189985 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qq5qd" event={"ID":"753d0899-a6ac-4059-befa-507d10de4609","Type":"ContainerDied","Data":"c2ccec23eaa659b90d7aa429359eb4c54afa5edd27077c8fbd7e311f9a92aa6f"} Jul 29 21:30:04.190203 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.190060 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2ccec23eaa659b90d7aa429359eb4c54afa5edd27077c8fbd7e311f9a92aa6f" Jul 29 21:30:04.193580 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.191568 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r" event={"ID":"e03589c3-e519-49dd-9fa6-5abd81122038","Type":"ContainerDied","Data":"65b3cf2c702c08c34f2af7b006060816cabed14b55b939083fcdabc771736f70"} Jul 29 21:30:04.193580 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.191610 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r" Jul 29 21:30:04.193580 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.191614 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65b3cf2c702c08c34f2af7b006060816cabed14b55b939083fcdabc771736f70" Jul 29 21:30:04.202994 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.202968 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c"] Jul 29 21:30:04.205282 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:30:04.205254 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99862056_cb0c_4c8b_a894_99947b14043e.slice/crio-864f96423f500818d097e2ba3f4404ba1f2da63acdabadef5e403ae49720f934 WatchSource:0}: Error finding container 864f96423f500818d097e2ba3f4404ba1f2da63acdabadef5e403ae49720f934: Status 404 returned error can't find the container with id 864f96423f500818d097e2ba3f4404ba1f2da63acdabadef5e403ae49720f934 Jul 29 21:30:04.207522 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.207490 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:30:04.207603 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.207553 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:30:04.207603 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:04.207589 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:30:05.198291 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:05.198252 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" event={"ID":"99862056-cb0c-4c8b-a894-99947b14043e","Type":"ContainerStarted","Data":"e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745"} Jul 29 21:30:05.198807 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:05.198313 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" event={"ID":"99862056-cb0c-4c8b-a894-99947b14043e","Type":"ContainerStarted","Data":"864f96423f500818d097e2ba3f4404ba1f2da63acdabadef5e403ae49720f934"} Jul 29 21:30:05.224893 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:05.224828 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" podStartSLOduration=2.224810476 podStartE2EDuration="2.224810476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:30:05.222598614 +0000 UTC m=+2553.346188457" watchObservedRunningTime="2026-07-29 21:30:05.224810476 +0000 UTC m=+2553.348400385" Jul 29 21:30:06.056308 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:06.056261 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:06.061642 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:06.061613 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:06.203941 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:06.203901 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:06.204909 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:06.204892 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:09.514725 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:09.514678 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk"] Jul 29 21:30:09.518010 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:09.517990 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" Jul 29 21:30:09.522150 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:09.522128 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3b326f-dockercfg-mmd6n\"" Jul 29 21:30:09.522402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:09.522384 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3b326f\"" Jul 29 21:30:09.530437 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:09.530413 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk"] Jul 29 21:30:09.648722 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:09.648687 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pzvp\" (UniqueName: \"kubernetes.io/projected/f1ec5218-29df-4400-a1d8-5b5117fad6a7-kube-api-access-8pzvp\") pod \"payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk\" (UID: \"f1ec5218-29df-4400-a1d8-5b5117fad6a7\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" Jul 29 21:30:09.648916 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:09.648798 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ec5218-29df-4400-a1d8-5b5117fad6a7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk\" (UID: \"f1ec5218-29df-4400-a1d8-5b5117fad6a7\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" Jul 29 21:30:09.749409 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:09.749375 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ec5218-29df-4400-a1d8-5b5117fad6a7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk\" (UID: \"f1ec5218-29df-4400-a1d8-5b5117fad6a7\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" Jul 29 21:30:09.749610 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:09.749475 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pzvp\" (UniqueName: \"kubernetes.io/projected/f1ec5218-29df-4400-a1d8-5b5117fad6a7-kube-api-access-8pzvp\") pod \"payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk\" (UID: \"f1ec5218-29df-4400-a1d8-5b5117fad6a7\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" Jul 29 21:30:09.750115 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:09.750096 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ec5218-29df-4400-a1d8-5b5117fad6a7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk\" (UID: \"f1ec5218-29df-4400-a1d8-5b5117fad6a7\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" Jul 29 21:30:09.758350 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:09.758322 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pzvp\" (UniqueName: \"kubernetes.io/projected/f1ec5218-29df-4400-a1d8-5b5117fad6a7-kube-api-access-8pzvp\") pod \"payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk\" (UID: \"f1ec5218-29df-4400-a1d8-5b5117fad6a7\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" Jul 29 21:30:09.830298 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:09.830192 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" Jul 29 21:30:10.190569 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:10.190546 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk"] Jul 29 21:30:10.191917 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:30:10.191887 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ec5218_29df_4400_a1d8_5b5117fad6a7.slice/crio-9fafc3738b84afde3c25bfd79d47a4ecbb58a6282ce114d0be7405d493af7a5e WatchSource:0}: Error finding container 9fafc3738b84afde3c25bfd79d47a4ecbb58a6282ce114d0be7405d493af7a5e: Status 404 returned error can't find the container with id 9fafc3738b84afde3c25bfd79d47a4ecbb58a6282ce114d0be7405d493af7a5e Jul 29 21:30:10.233043 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:10.232991 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" event={"ID":"f1ec5218-29df-4400-a1d8-5b5117fad6a7","Type":"ContainerStarted","Data":"9fafc3738b84afde3c25bfd79d47a4ecbb58a6282ce114d0be7405d493af7a5e"} Jul 29 21:30:11.240177 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:11.240137 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" event={"ID":"f1ec5218-29df-4400-a1d8-5b5117fad6a7","Type":"ContainerStarted","Data":"0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f"} Jul 29 21:30:11.240629 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:11.240272 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" Jul 29 21:30:11.260310 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:11.260255 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" podStartSLOduration=2.260237708 podStartE2EDuration="2.260237708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:30:11.257577448 +0000 UTC m=+2559.381167290" watchObservedRunningTime="2026-07-29 21:30:11.260237708 +0000 UTC m=+2559.383827590" Jul 29 21:30:14.559206 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:14.559167 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:30:22.247231 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:22.247181 2582 patch_prober.go:28] interesting pod/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 21:30:22.247828 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:22.247257 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" podUID="99862056-cb0c-4c8b-a894-99947b14043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 21:30:22.248696 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:22.248669 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" Jul 29 21:30:23.318567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:23.318531 2582 generic.go:356] "Generic (PLEG): container finished" podID="99862056-cb0c-4c8b-a894-99947b14043e" containerID="e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745" exitCode=0 Jul 29 21:30:23.319068 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:23.318588 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" event={"ID":"99862056-cb0c-4c8b-a894-99947b14043e","Type":"ContainerDied","Data":"e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745"} Jul 29 21:30:23.319068 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:23.318997 2582 scope.go:117] "RemoveContainer" containerID="e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745" Jul 29 21:30:23.320125 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:23.320091 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:30:23.320239 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:23.320183 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:30:23.320303 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:23.320233 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:30:24.326637 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:24.326604 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" event={"ID":"99862056-cb0c-4c8b-a894-99947b14043e","Type":"ContainerStarted","Data":"058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8"} Jul 29 21:30:25.056405 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:25.056366 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:25.056642 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:25.056622 2582 patch_prober.go:28] interesting pod/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Jul 29 21:30:25.056690 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:25.056670 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" podUID="99862056-cb0c-4c8b-a894-99947b14043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Jul 29 21:30:26.056592 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:26.056555 2582 patch_prober.go:28] interesting pod/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Jul 29 21:30:26.056991 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:26.056615 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" podUID="99862056-cb0c-4c8b-a894-99947b14043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Jul 29 21:30:27.056307 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:27.056274 2582 patch_prober.go:28] interesting pod/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Jul 29 21:30:27.056498 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:27.056332 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" podUID="99862056-cb0c-4c8b-a894-99947b14043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Jul 29 21:30:28.056551 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:28.056515 2582 patch_prober.go:28] interesting pod/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Jul 29 21:30:28.056970 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:28.056574 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" podUID="99862056-cb0c-4c8b-a894-99947b14043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Jul 29 21:30:29.059743 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:29.059703 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:29.060267 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:29.060049 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:29.060688 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:29.060671 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:30:50.627695 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:50.627649 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r"] Jul 29 21:30:50.630977 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:50.630952 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a7e568-29756010-lz48r"] Jul 29 21:30:50.661077 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:50.661045 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv"] Jul 29 21:30:50.661359 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:50.661306 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" podUID="4085153f-1520-4d91-95b5-9bbbc33b0902" containerName="payload-processing" containerID="cri-o://d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588" gracePeriod=30 Jul 29 21:30:50.674666 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:50.674633 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x"] Jul 29 21:30:50.674906 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:50.674880 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" podUID="93a432cd-51be-4524-a0c6-e3425fbd3219" containerName="payload-pre-processing" containerID="cri-o://066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429" gracePeriod=30 Jul 29 21:30:50.937957 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:50.937932 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" Jul 29 21:30:50.969062 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:50.969010 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" Jul 29 21:30:51.067692 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.067661 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a432cd-51be-4524-a0c6-e3425fbd3219-plugins-config-volume\") pod \"93a432cd-51be-4524-a0c6-e3425fbd3219\" (UID: \"93a432cd-51be-4524-a0c6-e3425fbd3219\") " Jul 29 21:30:51.067900 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.067732 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4085153f-1520-4d91-95b5-9bbbc33b0902-plugins-config-volume\") pod \"4085153f-1520-4d91-95b5-9bbbc33b0902\" (UID: \"4085153f-1520-4d91-95b5-9bbbc33b0902\") " Jul 29 21:30:51.067900 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.067781 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84stn\" (UniqueName: \"kubernetes.io/projected/4085153f-1520-4d91-95b5-9bbbc33b0902-kube-api-access-84stn\") pod \"4085153f-1520-4d91-95b5-9bbbc33b0902\" (UID: \"4085153f-1520-4d91-95b5-9bbbc33b0902\") " Jul 29 21:30:51.067900 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.067832 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmkvc\" (UniqueName: \"kubernetes.io/projected/93a432cd-51be-4524-a0c6-e3425fbd3219-kube-api-access-xmkvc\") pod \"93a432cd-51be-4524-a0c6-e3425fbd3219\" (UID: \"93a432cd-51be-4524-a0c6-e3425fbd3219\") " Jul 29 21:30:51.068205 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.068174 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a432cd-51be-4524-a0c6-e3425fbd3219-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "93a432cd-51be-4524-a0c6-e3425fbd3219" (UID: "93a432cd-51be-4524-a0c6-e3425fbd3219"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:30:51.068205 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.068196 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4085153f-1520-4d91-95b5-9bbbc33b0902-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4085153f-1520-4d91-95b5-9bbbc33b0902" (UID: "4085153f-1520-4d91-95b5-9bbbc33b0902"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:30:51.070105 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.070077 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4085153f-1520-4d91-95b5-9bbbc33b0902-kube-api-access-84stn" (OuterVolumeSpecName: "kube-api-access-84stn") pod "4085153f-1520-4d91-95b5-9bbbc33b0902" (UID: "4085153f-1520-4d91-95b5-9bbbc33b0902"). InnerVolumeSpecName "kube-api-access-84stn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:30:51.070197 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.070100 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a432cd-51be-4524-a0c6-e3425fbd3219-kube-api-access-xmkvc" (OuterVolumeSpecName: "kube-api-access-xmkvc") pod "93a432cd-51be-4524-a0c6-e3425fbd3219" (UID: "93a432cd-51be-4524-a0c6-e3425fbd3219"). InnerVolumeSpecName "kube-api-access-xmkvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:30:51.169513 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.169422 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84stn\" (UniqueName: \"kubernetes.io/projected/4085153f-1520-4d91-95b5-9bbbc33b0902-kube-api-access-84stn\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:30:51.169513 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.169453 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmkvc\" (UniqueName: \"kubernetes.io/projected/93a432cd-51be-4524-a0c6-e3425fbd3219-kube-api-access-xmkvc\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:30:51.169513 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.169463 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a432cd-51be-4524-a0c6-e3425fbd3219-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:30:51.169513 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.169471 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4085153f-1520-4d91-95b5-9bbbc33b0902-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:30:51.481957 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.481853 2582 generic.go:356] "Generic (PLEG): container finished" podID="93a432cd-51be-4524-a0c6-e3425fbd3219" containerID="066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429" exitCode=0 Jul 29 21:30:51.481957 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.481903 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" Jul 29 21:30:51.481957 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.481930 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" event={"ID":"93a432cd-51be-4524-a0c6-e3425fbd3219","Type":"ContainerDied","Data":"066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429"} Jul 29 21:30:51.482291 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.481996 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x" event={"ID":"93a432cd-51be-4524-a0c6-e3425fbd3219","Type":"ContainerDied","Data":"665e358a8b2b9418bf69d75fc304028b851187743f29c747c21661156d5eda06"} Jul 29 21:30:51.482291 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.482057 2582 scope.go:117] "RemoveContainer" containerID="066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429" Jul 29 21:30:51.483409 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.483388 2582 generic.go:356] "Generic (PLEG): container finished" podID="4085153f-1520-4d91-95b5-9bbbc33b0902" containerID="d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588" exitCode=0 Jul 29 21:30:51.483513 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.483438 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" Jul 29 21:30:51.483577 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.483441 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" event={"ID":"4085153f-1520-4d91-95b5-9bbbc33b0902","Type":"ContainerDied","Data":"d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588"} Jul 29 21:30:51.483639 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.483575 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv" event={"ID":"4085153f-1520-4d91-95b5-9bbbc33b0902","Type":"ContainerDied","Data":"0cff51f1b7d1f395ab8bc3e74350009e52a641d439d7d2ead04eeb71272fa696"} Jul 29 21:30:51.491718 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.491693 2582 scope.go:117] "RemoveContainer" containerID="066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429" Jul 29 21:30:51.492111 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:30:51.492086 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429\": container with ID starting with 066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429 not found: ID does not exist" containerID="066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429" Jul 29 21:30:51.492217 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.492121 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429"} err="failed to get container status \"066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429\": rpc error: code = NotFound desc = could not find container \"066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429\": container with ID starting with 066dec8ab2de8916c5568bc273f1daff9964d7819394b810019c0bb31a3ed429 not found: ID does not exist" Jul 29 21:30:51.492217 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.492143 2582 scope.go:117] "RemoveContainer" containerID="d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588" Jul 29 21:30:51.509622 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.509599 2582 scope.go:117] "RemoveContainer" containerID="d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588" Jul 29 21:30:51.509970 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:30:51.509940 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588\": container with ID starting with d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588 not found: ID does not exist" containerID="d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588" Jul 29 21:30:51.510046 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.509984 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588"} err="failed to get container status \"d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588\": rpc error: code = NotFound desc = could not find container \"d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588\": container with ID starting with d47b61e74ee9d9447748fa4944f1a27aad92774c747459fb1fd29cefecf9c588 not found: ID does not exist" Jul 29 21:30:51.514584 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.514557 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv"] Jul 29 21:30:51.518533 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.518508 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a7e568-77fcf58988-5vkzv"] Jul 29 21:30:51.529607 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.529572 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x"] Jul 29 21:30:51.533713 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:51.533687 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7e568-d76948c76-hhd2x"] Jul 29 21:30:52.493787 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:52.493745 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4085153f-1520-4d91-95b5-9bbbc33b0902" path="/var/lib/kubelet/pods/4085153f-1520-4d91-95b5-9bbbc33b0902/volumes" Jul 29 21:30:52.494429 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:52.494133 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93a432cd-51be-4524-a0c6-e3425fbd3219" path="/var/lib/kubelet/pods/93a432cd-51be-4524-a0c6-e3425fbd3219/volumes" Jul 29 21:30:52.494429 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:52.494373 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e03589c3-e519-49dd-9fa6-5abd81122038" path="/var/lib/kubelet/pods/e03589c3-e519-49dd-9fa6-5abd81122038/volumes" Jul 29 21:30:53.064634 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:53.064559 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:30:57.242049 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:57.241457 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz"] Jul 29 21:30:57.242049 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:30:57.241805 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" containerID="cri-o://35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e" gracePeriod=30 Jul 29 21:31:00.331882 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:00.331846 2582 patch_prober.go:28] interesting pod/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 21:31:00.332314 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:00.331917 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 21:31:02.502208 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.502180 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:31:02.550037 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.549933 2582 generic.go:356] "Generic (PLEG): container finished" podID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerID="35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e" exitCode=0 Jul 29 21:31:02.550037 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.550010 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" Jul 29 21:31:02.550293 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.550018 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" event={"ID":"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65","Type":"ContainerDied","Data":"35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e"} Jul 29 21:31:02.550293 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.550107 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz" event={"ID":"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65","Type":"ContainerDied","Data":"c190a9a0bc5f3c619c141e3e844c2cb19bd339f26bcd2ff0ef96ca050108169b"} Jul 29 21:31:02.550293 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.550138 2582 scope.go:117] "RemoveContainer" containerID="35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e" Jul 29 21:31:02.559533 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.559515 2582 scope.go:117] "RemoveContainer" containerID="0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12" Jul 29 21:31:02.567877 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.567845 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-token\") pod \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " Jul 29 21:31:02.567998 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.567881 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-podinfo\") pod \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " Jul 29 21:31:02.567998 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.567930 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-workload-certs\") pod \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " Jul 29 21:31:02.567998 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.567955 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-envoy\") pod \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " Jul 29 21:31:02.568247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.568001 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-data\") pod \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " Jul 29 21:31:02.568247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.568017 2582 scope.go:117] "RemoveContainer" containerID="35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e" Jul 29 21:31:02.568247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.568057 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxh5b\" (UniqueName: \"kubernetes.io/projected/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-kube-api-access-qxh5b\") pod \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " Jul 29 21:31:02.568247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.568125 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-credential-socket\") pod \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " Jul 29 21:31:02.568247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.568174 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istiod-ca-cert\") pod \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " Jul 29 21:31:02.568247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.568244 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-workload-socket\") pod \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\" (UID: \"2ee86d0f-e718-4b8a-84fe-3ceb09db5a65\") " Jul 29 21:31:02.569741 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:31:02.568698 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e\": container with ID starting with 35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e not found: ID does not exist" containerID="35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e" Jul 29 21:31:02.569741 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.568743 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e"} err="failed to get container status \"35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e\": rpc error: code = NotFound desc = could not find container \"35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e\": container with ID starting with 35b687773a1442193ce15d2233be8c451c454c99c1859ae5c7389d8a0416d80e not found: ID does not exist" Jul 29 21:31:02.569741 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.568770 2582 scope.go:117] "RemoveContainer" containerID="0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12" Jul 29 21:31:02.569741 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.569060 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" (UID: "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:31:02.569741 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.569075 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" (UID: "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:31:02.569741 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.569303 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" (UID: "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:31:02.569741 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.569586 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" (UID: "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:31:02.569741 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:31:02.569634 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12\": container with ID starting with 0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12 not found: ID does not exist" containerID="0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12" Jul 29 21:31:02.570352 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.569760 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12"} err="failed to get container status \"0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12\": rpc error: code = NotFound desc = could not find container \"0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12\": container with ID starting with 0ddd0834e639dec4bafaa0df4d1987d6cb2540f8b08f6f14ee9009ba4cd97e12 not found: ID does not exist" Jul 29 21:31:02.570352 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.569831 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" (UID: "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:31:02.570486 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.570463 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" (UID: "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:31:02.570612 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.570589 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" (UID: "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:31:02.570713 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.570688 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" (UID: "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:31:02.571171 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.571148 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-kube-api-access-qxh5b" (OuterVolumeSpecName: "kube-api-access-qxh5b") pod "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" (UID: "2ee86d0f-e718-4b8a-84fe-3ceb09db5a65"). InnerVolumeSpecName "kube-api-access-qxh5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:31:02.669250 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.669214 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:02.669250 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.669244 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:02.669250 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.669253 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:02.669497 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.669261 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:02.669497 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.669270 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:02.669497 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.669277 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:02.669497 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.669287 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxh5b\" (UniqueName: \"kubernetes.io/projected/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-kube-api-access-qxh5b\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:02.669497 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.669295 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:02.669497 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.669303 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:02.876513 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.876471 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz"] Jul 29 21:31:02.881577 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:02.881543 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a7e568-openshift-default-844c68764d-mxxwz"] Jul 29 21:31:04.492811 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:04.492776 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" path="/var/lib/kubelet/pods/2ee86d0f-e718-4b8a-84fe-3ceb09db5a65/volumes" Jul 29 21:31:13.108369 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.108327 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk"] Jul 29 21:31:13.108791 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.108620 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" podUID="f1ec5218-29df-4400-a1d8-5b5117fad6a7" containerName="payload-processing" containerID="cri-o://0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f" gracePeriod=30 Jul 29 21:31:13.371657 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.371593 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" Jul 29 21:31:13.463716 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.463675 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pzvp\" (UniqueName: \"kubernetes.io/projected/f1ec5218-29df-4400-a1d8-5b5117fad6a7-kube-api-access-8pzvp\") pod \"f1ec5218-29df-4400-a1d8-5b5117fad6a7\" (UID: \"f1ec5218-29df-4400-a1d8-5b5117fad6a7\") " Jul 29 21:31:13.463923 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.463797 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ec5218-29df-4400-a1d8-5b5117fad6a7-plugins-config-volume\") pod \"f1ec5218-29df-4400-a1d8-5b5117fad6a7\" (UID: \"f1ec5218-29df-4400-a1d8-5b5117fad6a7\") " Jul 29 21:31:13.464196 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.464169 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ec5218-29df-4400-a1d8-5b5117fad6a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f1ec5218-29df-4400-a1d8-5b5117fad6a7" (UID: "f1ec5218-29df-4400-a1d8-5b5117fad6a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:31:13.466043 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.465991 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ec5218-29df-4400-a1d8-5b5117fad6a7-kube-api-access-8pzvp" (OuterVolumeSpecName: "kube-api-access-8pzvp") pod "f1ec5218-29df-4400-a1d8-5b5117fad6a7" (UID: "f1ec5218-29df-4400-a1d8-5b5117fad6a7"). InnerVolumeSpecName "kube-api-access-8pzvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:31:13.564405 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.564370 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ec5218-29df-4400-a1d8-5b5117fad6a7-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:13.564405 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.564400 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pzvp\" (UniqueName: \"kubernetes.io/projected/f1ec5218-29df-4400-a1d8-5b5117fad6a7-kube-api-access-8pzvp\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:13.609694 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.609664 2582 generic.go:356] "Generic (PLEG): container finished" podID="f1ec5218-29df-4400-a1d8-5b5117fad6a7" containerID="0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f" exitCode=0 Jul 29 21:31:13.609927 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.609706 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" event={"ID":"f1ec5218-29df-4400-a1d8-5b5117fad6a7","Type":"ContainerDied","Data":"0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f"} Jul 29 21:31:13.609927 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.609714 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" Jul 29 21:31:13.609927 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.609757 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk" event={"ID":"f1ec5218-29df-4400-a1d8-5b5117fad6a7","Type":"ContainerDied","Data":"9fafc3738b84afde3c25bfd79d47a4ecbb58a6282ce114d0be7405d493af7a5e"} Jul 29 21:31:13.609927 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.609779 2582 scope.go:117] "RemoveContainer" containerID="0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f" Jul 29 21:31:13.619440 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.619423 2582 scope.go:117] "RemoveContainer" containerID="0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f" Jul 29 21:31:13.622635 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:31:13.622548 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f\": container with ID starting with 0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f not found: ID does not exist" containerID="0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f" Jul 29 21:31:13.622635 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.622588 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f"} err="failed to get container status \"0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f\": rpc error: code = NotFound desc = could not find container \"0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f\": container with ID starting with 0dd8067ce40afcd1d58873801146b83e290b77e7b21dac3ed97dbd5186b6846f not found: ID does not exist" Jul 29 21:31:13.635619 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.635590 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk"] Jul 29 21:31:13.639859 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:13.639838 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b326f-784bf5d4c4-p8nhk"] Jul 29 21:31:14.493621 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:14.493586 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1ec5218-29df-4400-a1d8-5b5117fad6a7" path="/var/lib/kubelet/pods/f1ec5218-29df-4400-a1d8-5b5117fad6a7/volumes" Jul 29 21:31:17.953079 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:17.953038 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:31:19.875401 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:19.875365 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c"] Jul 29 21:31:19.875860 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:19.875582 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" podUID="99862056-cb0c-4c8b-a894-99947b14043e" containerName="istio-proxy" containerID="cri-o://058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8" gracePeriod=30 Jul 29 21:31:20.818610 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.818573 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms"] Jul 29 21:31:20.818940 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.818928 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" Jul 29 21:31:20.818984 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.818942 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" Jul 29 21:31:20.818984 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.818953 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4085153f-1520-4d91-95b5-9bbbc33b0902" containerName="payload-processing" Jul 29 21:31:20.818984 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.818959 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4085153f-1520-4d91-95b5-9bbbc33b0902" containerName="payload-processing" Jul 29 21:31:20.818984 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.818970 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" Jul 29 21:31:20.818984 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.818975 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" Jul 29 21:31:20.818984 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.818982 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1ec5218-29df-4400-a1d8-5b5117fad6a7" containerName="payload-processing" Jul 29 21:31:20.819190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.818987 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1ec5218-29df-4400-a1d8-5b5117fad6a7" containerName="payload-processing" Jul 29 21:31:20.819190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.818996 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a432cd-51be-4524-a0c6-e3425fbd3219" containerName="payload-pre-processing" Jul 29 21:31:20.819190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.819001 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a432cd-51be-4524-a0c6-e3425fbd3219" containerName="payload-pre-processing" Jul 29 21:31:20.819190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.819071 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4085153f-1520-4d91-95b5-9bbbc33b0902" containerName="payload-processing" Jul 29 21:31:20.819190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.819080 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" Jul 29 21:31:20.819190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.819087 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee86d0f-e718-4b8a-84fe-3ceb09db5a65" containerName="istio-proxy" Jul 29 21:31:20.819190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.819092 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="93a432cd-51be-4524-a0c6-e3425fbd3219" containerName="payload-pre-processing" Jul 29 21:31:20.819190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.819098 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1ec5218-29df-4400-a1d8-5b5117fad6a7" containerName="payload-processing" Jul 29 21:31:20.822283 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.822257 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:20.824818 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.824797 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-eb3087-openshift-default-dockercfg-8m5wd\"" Jul 29 21:31:20.834153 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.834130 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms"] Jul 29 21:31:20.929227 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.929184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-istio-envoy\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:20.929227 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.929228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171461e2-b759-4116-992b-14d0feb8005f-istio-podinfo\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:20.929807 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.929261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171461e2-b759-4116-992b-14d0feb8005f-istio-token\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:20.929807 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.929352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-workload-socket\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:20.929807 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.929403 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-istio-data\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:20.929807 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.929448 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171461e2-b759-4116-992b-14d0feb8005f-istiod-ca-cert\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:20.929807 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.929474 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmwsb\" (UniqueName: \"kubernetes.io/projected/171461e2-b759-4116-992b-14d0feb8005f-kube-api-access-xmwsb\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:20.929807 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.929577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-workload-certs\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:20.929807 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:20.929608 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-credential-socket\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.030338 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.030302 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-workload-certs\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.030338 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.030342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-credential-socket\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.030596 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.030369 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-istio-envoy\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.030596 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.030388 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171461e2-b759-4116-992b-14d0feb8005f-istio-podinfo\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.030596 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.030418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171461e2-b759-4116-992b-14d0feb8005f-istio-token\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.030596 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.030446 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-workload-socket\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.030596 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.030467 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-istio-data\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.030596 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.030501 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171461e2-b759-4116-992b-14d0feb8005f-istiod-ca-cert\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.030911 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.030824 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-workload-certs\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.030911 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.030842 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-credential-socket\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.031088 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.030927 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-istio-data\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.031088 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.030525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmwsb\" (UniqueName: \"kubernetes.io/projected/171461e2-b759-4116-992b-14d0feb8005f-kube-api-access-xmwsb\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.031201 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.031114 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-workload-socket\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.031427 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.031238 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171461e2-b759-4116-992b-14d0feb8005f-istiod-ca-cert\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.032969 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.032940 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-istio-envoy\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.033193 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.033171 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171461e2-b759-4116-992b-14d0feb8005f-istio-podinfo\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.039389 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.039362 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmwsb\" (UniqueName: \"kubernetes.io/projected/171461e2-b759-4116-992b-14d0feb8005f-kube-api-access-xmwsb\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.039488 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.039398 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171461e2-b759-4116-992b-14d0feb8005f-istio-token\") pod \"e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.137981 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.137881 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:21.283895 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.283870 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms"] Jul 29 21:31:21.286546 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:31:21.286511 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171461e2_b759_4116_992b_14d0feb8005f.slice/crio-793f39a49f542120cc63849b1a7e218b98d0d19774c09ffa88140a9eea68dcbc WatchSource:0}: Error finding container 793f39a49f542120cc63849b1a7e218b98d0d19774c09ffa88140a9eea68dcbc: Status 404 returned error can't find the container with id 793f39a49f542120cc63849b1a7e218b98d0d19774c09ffa88140a9eea68dcbc Jul 29 21:31:21.288838 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.288796 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:31:21.288926 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.288887 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:31:21.288984 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.288934 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:31:21.657811 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.657774 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" event={"ID":"171461e2-b759-4116-992b-14d0feb8005f","Type":"ContainerStarted","Data":"10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308"} Jul 29 21:31:21.657996 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.657832 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" event={"ID":"171461e2-b759-4116-992b-14d0feb8005f","Type":"ContainerStarted","Data":"793f39a49f542120cc63849b1a7e218b98d0d19774c09ffa88140a9eea68dcbc"} Jul 29 21:31:21.679285 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:21.679236 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" podStartSLOduration=1.6792227830000002 podStartE2EDuration="1.679222783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:31:21.677755558 +0000 UTC m=+2629.801345398" watchObservedRunningTime="2026-07-29 21:31:21.679222783 +0000 UTC m=+2629.802812624" Jul 29 21:31:22.138089 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:22.138061 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:22.143330 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:22.143305 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:22.664448 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:22.664420 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:22.665664 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:22.665641 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:25.131891 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.131868 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:31:25.273125 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273087 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-workload-certs\") pod \"99862056-cb0c-4c8b-a894-99947b14043e\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " Jul 29 21:31:25.273301 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273135 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-istio-data\") pod \"99862056-cb0c-4c8b-a894-99947b14043e\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " Jul 29 21:31:25.273301 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273161 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-credential-socket\") pod \"99862056-cb0c-4c8b-a894-99947b14043e\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " Jul 29 21:31:25.273301 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273184 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99862056-cb0c-4c8b-a894-99947b14043e-istiod-ca-cert\") pod \"99862056-cb0c-4c8b-a894-99947b14043e\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " Jul 29 21:31:25.273301 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273206 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-istio-envoy\") pod \"99862056-cb0c-4c8b-a894-99947b14043e\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " Jul 29 21:31:25.273301 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273271 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99862056-cb0c-4c8b-a894-99947b14043e-istio-token\") pod \"99862056-cb0c-4c8b-a894-99947b14043e\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " Jul 29 21:31:25.273301 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273300 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-workload-socket\") pod \"99862056-cb0c-4c8b-a894-99947b14043e\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " Jul 29 21:31:25.273628 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273352 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2k4p\" (UniqueName: \"kubernetes.io/projected/99862056-cb0c-4c8b-a894-99947b14043e-kube-api-access-t2k4p\") pod \"99862056-cb0c-4c8b-a894-99947b14043e\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " Jul 29 21:31:25.273628 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273383 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99862056-cb0c-4c8b-a894-99947b14043e-istio-podinfo\") pod \"99862056-cb0c-4c8b-a894-99947b14043e\" (UID: \"99862056-cb0c-4c8b-a894-99947b14043e\") " Jul 29 21:31:25.273628 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273442 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99862056-cb0c-4c8b-a894-99947b14043e" (UID: "99862056-cb0c-4c8b-a894-99947b14043e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:31:25.273628 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273485 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99862056-cb0c-4c8b-a894-99947b14043e" (UID: "99862056-cb0c-4c8b-a894-99947b14043e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:31:25.273628 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273562 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99862056-cb0c-4c8b-a894-99947b14043e" (UID: "99862056-cb0c-4c8b-a894-99947b14043e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:31:25.273878 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273648 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99862056-cb0c-4c8b-a894-99947b14043e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99862056-cb0c-4c8b-a894-99947b14043e" (UID: "99862056-cb0c-4c8b-a894-99947b14043e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:31:25.273878 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273783 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:25.273878 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273802 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:25.273878 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273817 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99862056-cb0c-4c8b-a894-99947b14043e-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:25.273878 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273831 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:25.274353 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.273947 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-istio-data" (OuterVolumeSpecName: "istio-data") pod "99862056-cb0c-4c8b-a894-99947b14043e" (UID: "99862056-cb0c-4c8b-a894-99947b14043e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:31:25.275861 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.275808 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99862056-cb0c-4c8b-a894-99947b14043e-kube-api-access-t2k4p" (OuterVolumeSpecName: "kube-api-access-t2k4p") pod "99862056-cb0c-4c8b-a894-99947b14043e" (UID: "99862056-cb0c-4c8b-a894-99947b14043e"). InnerVolumeSpecName "kube-api-access-t2k4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:31:25.276011 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.275860 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99862056-cb0c-4c8b-a894-99947b14043e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99862056-cb0c-4c8b-a894-99947b14043e" (UID: "99862056-cb0c-4c8b-a894-99947b14043e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:31:25.276011 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.275907 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99862056-cb0c-4c8b-a894-99947b14043e-istio-token" (OuterVolumeSpecName: "istio-token") pod "99862056-cb0c-4c8b-a894-99947b14043e" (UID: "99862056-cb0c-4c8b-a894-99947b14043e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:31:25.276011 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.275938 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99862056-cb0c-4c8b-a894-99947b14043e" (UID: "99862056-cb0c-4c8b-a894-99947b14043e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:31:25.374476 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.374439 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2k4p\" (UniqueName: \"kubernetes.io/projected/99862056-cb0c-4c8b-a894-99947b14043e-kube-api-access-t2k4p\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:25.374476 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.374470 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99862056-cb0c-4c8b-a894-99947b14043e-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:25.374476 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.374479 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:25.374705 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.374487 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99862056-cb0c-4c8b-a894-99947b14043e-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:25.374705 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.374495 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99862056-cb0c-4c8b-a894-99947b14043e-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:31:25.484334 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.484298 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9"] Jul 29 21:31:25.484691 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.484678 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99862056-cb0c-4c8b-a894-99947b14043e" containerName="istio-proxy" Jul 29 21:31:25.484736 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.484693 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="99862056-cb0c-4c8b-a894-99947b14043e" containerName="istio-proxy" Jul 29 21:31:25.484736 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.484703 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99862056-cb0c-4c8b-a894-99947b14043e" containerName="istio-proxy" Jul 29 21:31:25.484736 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.484708 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="99862056-cb0c-4c8b-a894-99947b14043e" containerName="istio-proxy" Jul 29 21:31:25.484836 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.484774 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="99862056-cb0c-4c8b-a894-99947b14043e" containerName="istio-proxy" Jul 29 21:31:25.484836 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.484783 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="99862056-cb0c-4c8b-a894-99947b14043e" containerName="istio-proxy" Jul 29 21:31:25.487354 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.487333 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" Jul 29 21:31:25.490567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.490536 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-eb3087\"" Jul 29 21:31:25.490703 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.490536 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-eb3087-dockercfg-7hlsx\"" Jul 29 21:31:25.505795 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.505762 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9"] Jul 29 21:31:25.541705 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.541601 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd"] Jul 29 21:31:25.544245 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.544220 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" Jul 29 21:31:25.557999 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.557973 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd"] Jul 29 21:31:25.576613 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.576578 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nxqb\" (UniqueName: \"kubernetes.io/projected/6bcaed3c-f027-4b54-9ea7-cda91977fd92-kube-api-access-8nxqb\") pod \"payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9\" (UID: \"6bcaed3c-f027-4b54-9ea7-cda91977fd92\") " pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" Jul 29 21:31:25.576851 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.576666 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcaed3c-f027-4b54-9ea7-cda91977fd92-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9\" (UID: \"6bcaed3c-f027-4b54-9ea7-cda91977fd92\") " pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" Jul 29 21:31:25.677505 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.677472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nxqb\" (UniqueName: \"kubernetes.io/projected/6bcaed3c-f027-4b54-9ea7-cda91977fd92-kube-api-access-8nxqb\") pod \"payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9\" (UID: \"6bcaed3c-f027-4b54-9ea7-cda91977fd92\") " pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" Jul 29 21:31:25.677699 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.677518 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bb0446-62f0-435d-bc3e-580f02c6cf8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd\" (UID: \"b7bb0446-62f0-435d-bc3e-580f02c6cf8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" Jul 29 21:31:25.677699 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.677558 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtcwk\" (UniqueName: \"kubernetes.io/projected/b7bb0446-62f0-435d-bc3e-580f02c6cf8f-kube-api-access-gtcwk\") pod \"payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd\" (UID: \"b7bb0446-62f0-435d-bc3e-580f02c6cf8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" Jul 29 21:31:25.677699 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.677598 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcaed3c-f027-4b54-9ea7-cda91977fd92-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9\" (UID: \"6bcaed3c-f027-4b54-9ea7-cda91977fd92\") " pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" Jul 29 21:31:25.678331 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.678305 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcaed3c-f027-4b54-9ea7-cda91977fd92-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9\" (UID: \"6bcaed3c-f027-4b54-9ea7-cda91977fd92\") " pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" Jul 29 21:31:25.682318 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.682295 2582 generic.go:356] "Generic (PLEG): container finished" podID="99862056-cb0c-4c8b-a894-99947b14043e" containerID="058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8" exitCode=0 Jul 29 21:31:25.682417 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.682361 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" Jul 29 21:31:25.682417 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.682380 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" event={"ID":"99862056-cb0c-4c8b-a894-99947b14043e","Type":"ContainerDied","Data":"058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8"} Jul 29 21:31:25.682495 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.682447 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c" event={"ID":"99862056-cb0c-4c8b-a894-99947b14043e","Type":"ContainerDied","Data":"864f96423f500818d097e2ba3f4404ba1f2da63acdabadef5e403ae49720f934"} Jul 29 21:31:25.682495 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.682479 2582 scope.go:117] "RemoveContainer" containerID="058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8" Jul 29 21:31:25.686163 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.686140 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nxqb\" (UniqueName: \"kubernetes.io/projected/6bcaed3c-f027-4b54-9ea7-cda91977fd92-kube-api-access-8nxqb\") pod \"payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9\" (UID: \"6bcaed3c-f027-4b54-9ea7-cda91977fd92\") " pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" Jul 29 21:31:25.693000 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.692983 2582 scope.go:117] "RemoveContainer" containerID="e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745" Jul 29 21:31:25.702221 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.702191 2582 scope.go:117] "RemoveContainer" containerID="058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8" Jul 29 21:31:25.702510 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:31:25.702490 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8\": container with ID starting with 058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8 not found: ID does not exist" containerID="058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8" Jul 29 21:31:25.702564 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.702520 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8"} err="failed to get container status \"058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8\": rpc error: code = NotFound desc = could not find container \"058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8\": container with ID starting with 058e675961e5e0abee2a5d88bb32561d06810c1560049aea23a9c19f46a462f8 not found: ID does not exist" Jul 29 21:31:25.702564 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.702541 2582 scope.go:117] "RemoveContainer" containerID="e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745" Jul 29 21:31:25.702811 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:31:25.702789 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745\": container with ID starting with e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745 not found: ID does not exist" containerID="e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745" Jul 29 21:31:25.702854 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.702818 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745"} err="failed to get container status \"e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745\": rpc error: code = NotFound desc = could not find container \"e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745\": container with ID starting with e016d0b18da6655d2f80de385f35c3e16336b62808dea3aae4cf1a59647be745 not found: ID does not exist" Jul 29 21:31:25.707504 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.707480 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c"] Jul 29 21:31:25.711461 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.711441 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3b326f-openshift-default-d865b58fd-bl85c"] Jul 29 21:31:25.778863 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.778827 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcwk\" (UniqueName: \"kubernetes.io/projected/b7bb0446-62f0-435d-bc3e-580f02c6cf8f-kube-api-access-gtcwk\") pod \"payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd\" (UID: \"b7bb0446-62f0-435d-bc3e-580f02c6cf8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" Jul 29 21:31:25.779065 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.778974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bb0446-62f0-435d-bc3e-580f02c6cf8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd\" (UID: \"b7bb0446-62f0-435d-bc3e-580f02c6cf8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" Jul 29 21:31:25.779726 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.779706 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bb0446-62f0-435d-bc3e-580f02c6cf8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd\" (UID: \"b7bb0446-62f0-435d-bc3e-580f02c6cf8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" Jul 29 21:31:25.791207 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.791167 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcwk\" (UniqueName: \"kubernetes.io/projected/b7bb0446-62f0-435d-bc3e-580f02c6cf8f-kube-api-access-gtcwk\") pod \"payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd\" (UID: \"b7bb0446-62f0-435d-bc3e-580f02c6cf8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" Jul 29 21:31:25.801115 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.800987 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" Jul 29 21:31:25.854828 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.854786 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" Jul 29 21:31:25.975168 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:25.975141 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9"] Jul 29 21:31:25.976885 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:31:25.976849 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcaed3c_f027_4b54_9ea7_cda91977fd92.slice/crio-b081e04a9f7160d480d236f19bd015a5a8ad371414d96e74efc8e040e2169c6f WatchSource:0}: Error finding container b081e04a9f7160d480d236f19bd015a5a8ad371414d96e74efc8e040e2169c6f: Status 404 returned error can't find the container with id b081e04a9f7160d480d236f19bd015a5a8ad371414d96e74efc8e040e2169c6f Jul 29 21:31:26.040861 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:26.040832 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd"] Jul 29 21:31:26.042089 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:31:26.042062 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7bb0446_62f0_435d_bc3e_580f02c6cf8f.slice/crio-f21b8142396bf909d182a458170767a4aae1783b30d1ca5ec9db924584f3e2e5 WatchSource:0}: Error finding container f21b8142396bf909d182a458170767a4aae1783b30d1ca5ec9db924584f3e2e5: Status 404 returned error can't find the container with id f21b8142396bf909d182a458170767a4aae1783b30d1ca5ec9db924584f3e2e5 Jul 29 21:31:26.495270 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:26.495226 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99862056-cb0c-4c8b-a894-99947b14043e" path="/var/lib/kubelet/pods/99862056-cb0c-4c8b-a894-99947b14043e/volumes" Jul 29 21:31:26.690113 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:26.690075 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" event={"ID":"b7bb0446-62f0-435d-bc3e-580f02c6cf8f","Type":"ContainerStarted","Data":"dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848"} Jul 29 21:31:26.690327 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:26.690141 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" event={"ID":"b7bb0446-62f0-435d-bc3e-580f02c6cf8f","Type":"ContainerStarted","Data":"f21b8142396bf909d182a458170767a4aae1783b30d1ca5ec9db924584f3e2e5"} Jul 29 21:31:26.690327 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:26.690273 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" Jul 29 21:31:26.691645 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:26.691618 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" event={"ID":"6bcaed3c-f027-4b54-9ea7-cda91977fd92","Type":"ContainerStarted","Data":"6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0"} Jul 29 21:31:26.691799 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:26.691664 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" event={"ID":"6bcaed3c-f027-4b54-9ea7-cda91977fd92","Type":"ContainerStarted","Data":"b081e04a9f7160d480d236f19bd015a5a8ad371414d96e74efc8e040e2169c6f"} Jul 29 21:31:26.691799 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:26.691751 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" Jul 29 21:31:26.713040 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:26.712983 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" podStartSLOduration=1.712968684 podStartE2EDuration="1.712968684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:31:26.710573603 +0000 UTC m=+2634.834163448" watchObservedRunningTime="2026-07-29 21:31:26.712968684 +0000 UTC m=+2634.836558525" Jul 29 21:31:26.727591 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:26.727538 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" podStartSLOduration=1.727522998 podStartE2EDuration="1.727522998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:31:26.726746294 +0000 UTC m=+2634.850336135" watchObservedRunningTime="2026-07-29 21:31:26.727522998 +0000 UTC m=+2634.851112839" Jul 29 21:31:29.479652 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:29.479613 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:31:32.782132 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:32.782092 2582 scope.go:117] "RemoveContainer" containerID="3d7a2cc86d9dd741030a34e9d1c3fe8a32bd3ac217a8be2cda9d54e2482414e1" Jul 29 21:31:37.665981 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:37.665949 2582 patch_prober.go:28] interesting pod/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Jul 29 21:31:37.666456 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:37.666007 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Jul 29 21:31:37.700483 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:37.700451 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" Jul 29 21:31:37.700688 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:37.700526 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" Jul 29 21:31:37.764768 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:37.764739 2582 generic.go:356] "Generic (PLEG): container finished" podID="171461e2-b759-4116-992b-14d0feb8005f" containerID="10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308" exitCode=0 Jul 29 21:31:37.764940 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:37.764821 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" event={"ID":"171461e2-b759-4116-992b-14d0feb8005f","Type":"ContainerDied","Data":"10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308"} Jul 29 21:31:37.765358 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:37.765343 2582 scope.go:117] "RemoveContainer" containerID="10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308" Jul 29 21:31:37.766420 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:37.766379 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:31:37.766563 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:37.766473 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:31:37.766563 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:37.766524 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:31:38.771899 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:38.771861 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" event={"ID":"171461e2-b759-4116-992b-14d0feb8005f","Type":"ContainerStarted","Data":"8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb"} Jul 29 21:31:39.139194 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:39.139089 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:39.139414 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:39.139388 2582 patch_prober.go:28] interesting pod/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Jul 29 21:31:39.139500 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:39.139466 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Jul 29 21:31:40.139261 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:40.139225 2582 patch_prober.go:28] interesting pod/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Jul 29 21:31:40.139675 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:40.139296 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Jul 29 21:31:41.138474 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:41.138431 2582 patch_prober.go:28] interesting pod/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Jul 29 21:31:41.138671 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:41.138505 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Jul 29 21:31:42.139196 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:42.139159 2582 patch_prober.go:28] interesting pod/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Jul 29 21:31:42.139601 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:42.139212 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Jul 29 21:31:43.139105 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:43.139068 2582 patch_prober.go:28] interesting pod/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Jul 29 21:31:43.139303 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:43.139143 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Jul 29 21:31:44.143314 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:44.143284 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:44.143758 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:44.143653 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:44.144501 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:44.144485 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:31:49.266059 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.266005 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n"] Jul 29 21:31:49.269216 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.269196 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.272516 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.272495 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-46bb38-openshift-default-dockercfg-cdd4w\"" Jul 29 21:31:49.281299 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.281270 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n"] Jul 29 21:31:49.298427 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.298387 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-podinfo\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.298574 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.298472 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cr4m\" (UniqueName: \"kubernetes.io/projected/4360fb49-4d3c-497e-a4c3-22e4f05866de-kube-api-access-7cr4m\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.298574 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.298534 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-data\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.298574 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.298561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4360fb49-4d3c-497e-a4c3-22e4f05866de-istiod-ca-cert\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.298750 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.298603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-envoy\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.298750 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.298631 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-workload-socket\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.298750 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.298655 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-workload-certs\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.298750 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.298707 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-token\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.298750 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.298741 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-credential-socket\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.399709 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.399671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cr4m\" (UniqueName: \"kubernetes.io/projected/4360fb49-4d3c-497e-a4c3-22e4f05866de-kube-api-access-7cr4m\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.399919 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.399728 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-data\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.399919 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.399832 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4360fb49-4d3c-497e-a4c3-22e4f05866de-istiod-ca-cert\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.399919 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.399902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-envoy\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.400140 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.399940 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-workload-socket\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.400140 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.399964 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-workload-certs\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.400140 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.400004 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-token\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.400140 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.400052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-credential-socket\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.400140 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.400107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-podinfo\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.400140 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.400130 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-data\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.400460 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.400289 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-workload-socket\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.400460 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.400413 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-workload-certs\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.400567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.400461 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-credential-socket\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.400649 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.400629 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4360fb49-4d3c-497e-a4c3-22e4f05866de-istiod-ca-cert\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.402414 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.402386 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-envoy\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.402841 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.402820 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-podinfo\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.409534 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.409497 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-token\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.409717 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.409692 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cr4m\" (UniqueName: \"kubernetes.io/projected/4360fb49-4d3c-497e-a4c3-22e4f05866de-kube-api-access-7cr4m\") pod \"e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.580398 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.580304 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:49.733853 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.733826 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n"] Jul 29 21:31:49.735180 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:31:49.735151 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4360fb49_4d3c_497e_a4c3_22e4f05866de.slice/crio-70aadadf39518fedda9a10568b1beabcbc4b7fa077a59cd5cadab8b294a87e88 WatchSource:0}: Error finding container 70aadadf39518fedda9a10568b1beabcbc4b7fa077a59cd5cadab8b294a87e88: Status 404 returned error can't find the container with id 70aadadf39518fedda9a10568b1beabcbc4b7fa077a59cd5cadab8b294a87e88 Jul 29 21:31:49.737591 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.737546 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:31:49.737685 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.737643 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:31:49.737733 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.737689 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:31:49.834363 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.834272 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" event={"ID":"4360fb49-4d3c-497e-a4c3-22e4f05866de","Type":"ContainerStarted","Data":"bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692"} Jul 29 21:31:49.834363 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.834325 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" event={"ID":"4360fb49-4d3c-497e-a4c3-22e4f05866de","Type":"ContainerStarted","Data":"70aadadf39518fedda9a10568b1beabcbc4b7fa077a59cd5cadab8b294a87e88"} Jul 29 21:31:49.854640 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:49.854586 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" podStartSLOduration=0.854568843 podStartE2EDuration="854.568843ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:31:49.853183319 +0000 UTC m=+2657.976773175" watchObservedRunningTime="2026-07-29 21:31:49.854568843 +0000 UTC m=+2657.978158684" Jul 29 21:31:50.581449 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:50.581409 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:50.586666 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:50.586640 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:50.840392 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:50.840304 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:50.841264 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:50.841242 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:31:53.483331 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.483284 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52"] Jul 29 21:31:53.485982 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.485966 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" Jul 29 21:31:53.490255 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.490232 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-46bb38\"" Jul 29 21:31:53.490377 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.490290 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-46bb38-dockercfg-gcn58\"" Jul 29 21:31:53.500447 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.500422 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52"] Jul 29 21:31:53.505006 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.504982 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5"] Jul 29 21:31:53.507681 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.507664 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" Jul 29 21:31:53.521444 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.521415 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5"] Jul 29 21:31:53.542869 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.542839 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8hsj\" (UniqueName: \"kubernetes.io/projected/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b-kube-api-access-v8hsj\") pod \"payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52\" (UID: \"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" Jul 29 21:31:53.543061 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.542881 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a10363e-a2d1-437f-81c8-842810aad721-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5\" (UID: \"5a10363e-a2d1-437f-81c8-842810aad721\") " pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" Jul 29 21:31:53.543061 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.542963 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9l5k\" (UniqueName: \"kubernetes.io/projected/5a10363e-a2d1-437f-81c8-842810aad721-kube-api-access-r9l5k\") pod \"payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5\" (UID: \"5a10363e-a2d1-437f-81c8-842810aad721\") " pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" Jul 29 21:31:53.543061 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.542986 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52\" (UID: \"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" Jul 29 21:31:53.644145 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.644100 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9l5k\" (UniqueName: \"kubernetes.io/projected/5a10363e-a2d1-437f-81c8-842810aad721-kube-api-access-r9l5k\") pod \"payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5\" (UID: \"5a10363e-a2d1-437f-81c8-842810aad721\") " pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" Jul 29 21:31:53.644369 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.644162 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52\" (UID: \"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" Jul 29 21:31:53.644369 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.644219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8hsj\" (UniqueName: \"kubernetes.io/projected/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b-kube-api-access-v8hsj\") pod \"payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52\" (UID: \"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" Jul 29 21:31:53.644369 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.644262 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a10363e-a2d1-437f-81c8-842810aad721-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5\" (UID: \"5a10363e-a2d1-437f-81c8-842810aad721\") " pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" Jul 29 21:31:53.644901 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.644869 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52\" (UID: \"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" Jul 29 21:31:53.645098 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.645077 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a10363e-a2d1-437f-81c8-842810aad721-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5\" (UID: \"5a10363e-a2d1-437f-81c8-842810aad721\") " pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" Jul 29 21:31:53.659185 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.659161 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8hsj\" (UniqueName: \"kubernetes.io/projected/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b-kube-api-access-v8hsj\") pod \"payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52\" (UID: \"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" Jul 29 21:31:53.659341 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.659322 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9l5k\" (UniqueName: \"kubernetes.io/projected/5a10363e-a2d1-437f-81c8-842810aad721-kube-api-access-r9l5k\") pod \"payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5\" (UID: \"5a10363e-a2d1-437f-81c8-842810aad721\") " pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" Jul 29 21:31:53.799657 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.799611 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" Jul 29 21:31:53.819697 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.819666 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" Jul 29 21:31:53.960109 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.960083 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52"] Jul 29 21:31:53.962372 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:31:53.962325 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc345ed65_4a4f_4b61_ac8e_cf0daa593f6b.slice/crio-5a0327ed4ef6c96e751956d791bc623072577c130507dceed3e1d0cc51064c5b WatchSource:0}: Error finding container 5a0327ed4ef6c96e751956d791bc623072577c130507dceed3e1d0cc51064c5b: Status 404 returned error can't find the container with id 5a0327ed4ef6c96e751956d791bc623072577c130507dceed3e1d0cc51064c5b Jul 29 21:31:53.984718 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:53.984694 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5"] Jul 29 21:31:53.992359 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:31:53.992336 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a10363e_a2d1_437f_81c8_842810aad721.slice/crio-daf083ff5cf46a05ed9c09d7b80ac7503502c8ab5927e16639d5379d06959262 WatchSource:0}: Error finding container daf083ff5cf46a05ed9c09d7b80ac7503502c8ab5927e16639d5379d06959262: Status 404 returned error can't find the container with id daf083ff5cf46a05ed9c09d7b80ac7503502c8ab5927e16639d5379d06959262 Jul 29 21:31:54.864704 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:54.864671 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" event={"ID":"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b","Type":"ContainerStarted","Data":"5f112522c05bf6e85b465f003a5c7aa77648bbd1a5b4dc6097ea64907b13dcf6"} Jul 29 21:31:54.865264 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:54.864721 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" event={"ID":"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b","Type":"ContainerStarted","Data":"5a0327ed4ef6c96e751956d791bc623072577c130507dceed3e1d0cc51064c5b"} Jul 29 21:31:54.865264 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:54.864778 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" Jul 29 21:31:54.866426 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:54.866401 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" event={"ID":"5a10363e-a2d1-437f-81c8-842810aad721","Type":"ContainerStarted","Data":"71ae872f417b24ed27ac1183dbac146a0089d89df21d3280e6714e756a9b1c2e"} Jul 29 21:31:54.866542 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:54.866437 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" event={"ID":"5a10363e-a2d1-437f-81c8-842810aad721","Type":"ContainerStarted","Data":"daf083ff5cf46a05ed9c09d7b80ac7503502c8ab5927e16639d5379d06959262"} Jul 29 21:31:54.866542 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:54.866478 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" Jul 29 21:31:54.880905 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:54.880862 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" podStartSLOduration=1.8808512899999998 podStartE2EDuration="1.88085129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:31:54.879814162 +0000 UTC m=+2663.003404003" watchObservedRunningTime="2026-07-29 21:31:54.88085129 +0000 UTC m=+2663.004441132" Jul 29 21:31:54.896787 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:54.896735 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" podStartSLOduration=1.896717502 podStartE2EDuration="1.896717502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:31:54.895939179 +0000 UTC m=+2663.019529022" watchObservedRunningTime="2026-07-29 21:31:54.896717502 +0000 UTC m=+2663.020307335" Jul 29 21:31:58.964693 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:31:58.964658 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:32:05.874163 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:05.874126 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" Jul 29 21:32:05.874666 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:05.874195 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" Jul 29 21:32:06.881238 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:06.881195 2582 patch_prober.go:28] interesting pod/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 21:32:06.881630 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:06.881280 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 21:32:07.955939 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:07.955906 2582 generic.go:356] "Generic (PLEG): container finished" podID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerID="bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692" exitCode=0 Jul 29 21:32:07.956397 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:07.955960 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" event={"ID":"4360fb49-4d3c-497e-a4c3-22e4f05866de","Type":"ContainerDied","Data":"bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692"} Jul 29 21:32:07.956397 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:07.956373 2582 scope.go:117] "RemoveContainer" containerID="bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692" Jul 29 21:32:07.957417 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:07.957386 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:32:07.957474 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:07.957462 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:32:07.957511 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:07.957497 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:32:08.963486 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:08.963451 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" event={"ID":"4360fb49-4d3c-497e-a4c3-22e4f05866de","Type":"ContainerStarted","Data":"db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a"} Jul 29 21:32:09.580898 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:09.580862 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:32:09.581122 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:09.581101 2582 patch_prober.go:28] interesting pod/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Jul 29 21:32:09.581189 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:09.581168 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Jul 29 21:32:10.581511 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:10.581469 2582 patch_prober.go:28] interesting pod/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Jul 29 21:32:10.582011 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:10.581553 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Jul 29 21:32:11.580860 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:11.580815 2582 patch_prober.go:28] interesting pod/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Jul 29 21:32:11.581088 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:11.580881 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Jul 29 21:32:12.581253 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:12.581214 2582 patch_prober.go:28] interesting pod/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Jul 29 21:32:12.581647 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:12.581272 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Jul 29 21:32:13.584206 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:13.584176 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:32:13.584681 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:13.584454 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:32:13.585388 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:13.585371 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:32:40.650428 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:32:40.650389 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:33:04.756313 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:33:04.756273 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:33:28.355000 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:33:28.354960 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:33:56.353267 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:33:56.353185 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:34:12.166291 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:34:12.166250 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:34:39.654397 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:34:39.654357 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:34:57.653735 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:34:57.653694 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:35:24.180018 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:35:24.179930 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:35:32.863712 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:35:32.863679 2582 scope.go:117] "RemoveContainer" containerID="c97e08b5c2713aa10741d33bd13877720be8957edb671bd6df8ee4a4c5b71861" Jul 29 21:35:41.457153 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:35:41.457112 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:36:01.979463 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:36:01.979420 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:36:21.054382 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:36:21.054341 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:36:32.885633 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:36:32.885601 2582 scope.go:117] "RemoveContainer" containerID="693bf7c95ac327f98a579fd5a09c9b8d19bd88fd94d3278b1e54e55ee29b2ecb" Jul 29 21:36:47.563177 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:36:47.563140 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:37:05.365199 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:37:05.365154 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:37:32.807135 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:37:32.807099 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:37:48.261539 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:37:48.261500 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:37:59.850128 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:37:59.850095 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:38:40.953265 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:38:40.953179 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:39:11.659990 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:39:11.659954 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:39:31.063764 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:39:31.063730 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:39:48.569206 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:39:48.569160 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:39:58.768680 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:39:58.768586 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:40:17.160196 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:40:17.160165 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:40:56.318132 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:40:56.318097 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:41:12.006040 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:12.005997 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:41:26.977191 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:26.977107 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:41:33.771525 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:33.771489 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7477-139989194276144"] Jul 29 21:41:33.774868 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:33.774850 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194276144" Jul 29 21:41:33.778347 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:33.778327 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zmhw5\"" Jul 29 21:41:33.790354 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:33.790320 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-139989194276144"] Jul 29 21:41:33.824435 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:33.824389 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbmcl\" (UniqueName: \"kubernetes.io/projected/a74a306b-5cd6-4cb5-9c45-5eca2758dde5-kube-api-access-gbmcl\") pod \"test-curl-7477-139989194276144\" (UID: \"a74a306b-5cd6-4cb5-9c45-5eca2758dde5\") " pod="opendatahub/test-curl-7477-139989194276144" Jul 29 21:41:33.925003 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:33.924953 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbmcl\" (UniqueName: \"kubernetes.io/projected/a74a306b-5cd6-4cb5-9c45-5eca2758dde5-kube-api-access-gbmcl\") pod \"test-curl-7477-139989194276144\" (UID: \"a74a306b-5cd6-4cb5-9c45-5eca2758dde5\") " pod="opendatahub/test-curl-7477-139989194276144" Jul 29 21:41:33.936995 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:33.936972 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbmcl\" (UniqueName: \"kubernetes.io/projected/a74a306b-5cd6-4cb5-9c45-5eca2758dde5-kube-api-access-gbmcl\") pod \"test-curl-7477-139989194276144\" (UID: \"a74a306b-5cd6-4cb5-9c45-5eca2758dde5\") " pod="opendatahub/test-curl-7477-139989194276144" Jul 29 21:41:34.085043 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:34.084935 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194276144" Jul 29 21:41:34.241979 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:34.241956 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-139989194276144"] Jul 29 21:41:34.243985 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:41:34.243948 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda74a306b_5cd6_4cb5_9c45_5eca2758dde5.slice/crio-f0341f4b5db6f27a1a8f44bdfb490b83ca241913bf4795a8e0f448605ddf2492 WatchSource:0}: Error finding container f0341f4b5db6f27a1a8f44bdfb490b83ca241913bf4795a8e0f448605ddf2492: Status 404 returned error can't find the container with id f0341f4b5db6f27a1a8f44bdfb490b83ca241913bf4795a8e0f448605ddf2492 Jul 29 21:41:34.245347 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:34.245326 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 21:41:35.086101 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:35.086064 2582 generic.go:356] "Generic (PLEG): container finished" podID="a74a306b-5cd6-4cb5-9c45-5eca2758dde5" containerID="f5e94f67990f1c60908be1db5fa4c3bc16e29fb587a12bb4e05d8b453d7d1c89" exitCode=0 Jul 29 21:41:35.086101 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:35.086104 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-139989194276144" event={"ID":"a74a306b-5cd6-4cb5-9c45-5eca2758dde5","Type":"ContainerDied","Data":"f5e94f67990f1c60908be1db5fa4c3bc16e29fb587a12bb4e05d8b453d7d1c89"} Jul 29 21:41:35.086531 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:35.086135 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-139989194276144" event={"ID":"a74a306b-5cd6-4cb5-9c45-5eca2758dde5","Type":"ContainerStarted","Data":"f0341f4b5db6f27a1a8f44bdfb490b83ca241913bf4795a8e0f448605ddf2492"} Jul 29 21:41:36.232556 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.232531 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194276144" Jul 29 21:41:36.301808 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.301774 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7477-139989194276144_a74a306b-5cd6-4cb5-9c45-5eca2758dde5/test-curl-7477-139989194276144/0.log" Jul 29 21:41:36.340840 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.340802 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7477-139989194276144"] Jul 29 21:41:36.344346 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.344284 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7477-139989194276144"] Jul 29 21:41:36.347765 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.347742 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbmcl\" (UniqueName: \"kubernetes.io/projected/a74a306b-5cd6-4cb5-9c45-5eca2758dde5-kube-api-access-gbmcl\") pod \"a74a306b-5cd6-4cb5-9c45-5eca2758dde5\" (UID: \"a74a306b-5cd6-4cb5-9c45-5eca2758dde5\") " Jul 29 21:41:36.350131 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.350104 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74a306b-5cd6-4cb5-9c45-5eca2758dde5-kube-api-access-gbmcl" (OuterVolumeSpecName: "kube-api-access-gbmcl") pod "a74a306b-5cd6-4cb5-9c45-5eca2758dde5" (UID: "a74a306b-5cd6-4cb5-9c45-5eca2758dde5"). InnerVolumeSpecName "kube-api-access-gbmcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:41:36.449287 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.449257 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbmcl\" (UniqueName: \"kubernetes.io/projected/a74a306b-5cd6-4cb5-9c45-5eca2758dde5-kube-api-access-gbmcl\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:41:36.471599 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.471563 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7477-139989194257584"] Jul 29 21:41:36.471982 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.471967 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74a306b-5cd6-4cb5-9c45-5eca2758dde5" containerName="test-curl-7477-139989194276144" Jul 29 21:41:36.472067 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.471983 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74a306b-5cd6-4cb5-9c45-5eca2758dde5" containerName="test-curl-7477-139989194276144" Jul 29 21:41:36.472067 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.472057 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74a306b-5cd6-4cb5-9c45-5eca2758dde5" containerName="test-curl-7477-139989194276144" Jul 29 21:41:36.474325 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.474308 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194257584" Jul 29 21:41:36.496710 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.496672 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a74a306b-5cd6-4cb5-9c45-5eca2758dde5" path="/var/lib/kubelet/pods/a74a306b-5cd6-4cb5-9c45-5eca2758dde5/volumes" Jul 29 21:41:36.497227 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.497206 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-139989194257584"] Jul 29 21:41:36.549813 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.549783 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcqb4\" (UniqueName: \"kubernetes.io/projected/916082e2-1629-45e4-9b7a-8ac5f50078d8-kube-api-access-wcqb4\") pod \"test-curl-7477-139989194257584\" (UID: \"916082e2-1629-45e4-9b7a-8ac5f50078d8\") " pod="opendatahub/test-curl-7477-139989194257584" Jul 29 21:41:36.651057 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.650933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcqb4\" (UniqueName: \"kubernetes.io/projected/916082e2-1629-45e4-9b7a-8ac5f50078d8-kube-api-access-wcqb4\") pod \"test-curl-7477-139989194257584\" (UID: \"916082e2-1629-45e4-9b7a-8ac5f50078d8\") " pod="opendatahub/test-curl-7477-139989194257584" Jul 29 21:41:36.663526 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.663490 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcqb4\" (UniqueName: \"kubernetes.io/projected/916082e2-1629-45e4-9b7a-8ac5f50078d8-kube-api-access-wcqb4\") pod \"test-curl-7477-139989194257584\" (UID: \"916082e2-1629-45e4-9b7a-8ac5f50078d8\") " pod="opendatahub/test-curl-7477-139989194257584" Jul 29 21:41:36.783660 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.783625 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194257584" Jul 29 21:41:36.938391 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:36.938306 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-139989194257584"] Jul 29 21:41:36.941570 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:41:36.941545 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod916082e2_1629_45e4_9b7a_8ac5f50078d8.slice/crio-117a4adf583cd0a1c9aa3abb6d69b6d5acaffbb88fb50cd3acbfaf739027f1f8 WatchSource:0}: Error finding container 117a4adf583cd0a1c9aa3abb6d69b6d5acaffbb88fb50cd3acbfaf739027f1f8: Status 404 returned error can't find the container with id 117a4adf583cd0a1c9aa3abb6d69b6d5acaffbb88fb50cd3acbfaf739027f1f8 Jul 29 21:41:37.097813 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:37.097782 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194276144" Jul 29 21:41:37.098059 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:37.097792 2582 scope.go:117] "RemoveContainer" containerID="f5e94f67990f1c60908be1db5fa4c3bc16e29fb587a12bb4e05d8b453d7d1c89" Jul 29 21:41:37.099636 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:37.099602 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-139989194257584" event={"ID":"916082e2-1629-45e4-9b7a-8ac5f50078d8","Type":"ContainerStarted","Data":"117a4adf583cd0a1c9aa3abb6d69b6d5acaffbb88fb50cd3acbfaf739027f1f8"} Jul 29 21:41:38.106888 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:38.106855 2582 generic.go:356] "Generic (PLEG): container finished" podID="916082e2-1629-45e4-9b7a-8ac5f50078d8" containerID="0d98c1f2ac4f7906379046e5b10b0c051a2d7a9aee93dc42e68009e58fabc3d8" exitCode=0 Jul 29 21:41:38.107365 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:38.106954 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-139989194257584" event={"ID":"916082e2-1629-45e4-9b7a-8ac5f50078d8","Type":"ContainerDied","Data":"0d98c1f2ac4f7906379046e5b10b0c051a2d7a9aee93dc42e68009e58fabc3d8"} Jul 29 21:41:39.250979 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.250956 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194257584" Jul 29 21:41:39.275317 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.275273 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcqb4\" (UniqueName: \"kubernetes.io/projected/916082e2-1629-45e4-9b7a-8ac5f50078d8-kube-api-access-wcqb4\") pod \"916082e2-1629-45e4-9b7a-8ac5f50078d8\" (UID: \"916082e2-1629-45e4-9b7a-8ac5f50078d8\") " Jul 29 21:41:39.277785 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.277757 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916082e2-1629-45e4-9b7a-8ac5f50078d8-kube-api-access-wcqb4" (OuterVolumeSpecName: "kube-api-access-wcqb4") pod "916082e2-1629-45e4-9b7a-8ac5f50078d8" (UID: "916082e2-1629-45e4-9b7a-8ac5f50078d8"). InnerVolumeSpecName "kube-api-access-wcqb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:41:39.302286 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.302259 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7477-139989194257584_916082e2-1629-45e4-9b7a-8ac5f50078d8/test-curl-7477-139989194257584/0.log" Jul 29 21:41:39.342903 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.342875 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7477-139989194257584"] Jul 29 21:41:39.346564 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.346525 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7477-139989194257584"] Jul 29 21:41:39.376544 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.376464 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcqb4\" (UniqueName: \"kubernetes.io/projected/916082e2-1629-45e4-9b7a-8ac5f50078d8-kube-api-access-wcqb4\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:41:39.544278 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.544244 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7477-139989194131472"] Jul 29 21:41:39.544641 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.544629 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916082e2-1629-45e4-9b7a-8ac5f50078d8" containerName="test-curl-7477-139989194257584" Jul 29 21:41:39.544690 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.544643 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="916082e2-1629-45e4-9b7a-8ac5f50078d8" containerName="test-curl-7477-139989194257584" Jul 29 21:41:39.544728 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.544706 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="916082e2-1629-45e4-9b7a-8ac5f50078d8" containerName="test-curl-7477-139989194257584" Jul 29 21:41:39.546786 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.546770 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:39.566123 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.566093 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-139989194131472"] Jul 29 21:41:39.578688 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.578656 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgpv6\" (UniqueName: \"kubernetes.io/projected/f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253-kube-api-access-wgpv6\") pod \"test-curl-7477-139989194131472\" (UID: \"f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253\") " pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:39.679519 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.679420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgpv6\" (UniqueName: \"kubernetes.io/projected/f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253-kube-api-access-wgpv6\") pod \"test-curl-7477-139989194131472\" (UID: \"f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253\") " pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:39.695105 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.695071 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgpv6\" (UniqueName: \"kubernetes.io/projected/f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253-kube-api-access-wgpv6\") pod \"test-curl-7477-139989194131472\" (UID: \"f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253\") " pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:39.855979 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:39.855935 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:40.013540 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:40.013507 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-139989194131472"] Jul 29 21:41:40.014855 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:41:40.014832 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf41eeb19_a9bd_4abd_a47e_ba4c6d0b6253.slice/crio-af776747099f96749c582eb557a28929598dafb09c8018f86aa4a530c385c00e WatchSource:0}: Error finding container af776747099f96749c582eb557a28929598dafb09c8018f86aa4a530c385c00e: Status 404 returned error can't find the container with id af776747099f96749c582eb557a28929598dafb09c8018f86aa4a530c385c00e Jul 29 21:41:40.121180 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:40.121144 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-139989194131472" event={"ID":"f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253","Type":"ContainerStarted","Data":"af776747099f96749c582eb557a28929598dafb09c8018f86aa4a530c385c00e"} Jul 29 21:41:40.122532 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:40.122510 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="117a4adf583cd0a1c9aa3abb6d69b6d5acaffbb88fb50cd3acbfaf739027f1f8" Jul 29 21:41:40.122648 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:40.122563 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194257584" Jul 29 21:41:40.133500 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:41:40.133471 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7477-139989194257584\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="916082e2-1629-45e4-9b7a-8ac5f50078d8" pod="opendatahub/test-curl-7477-139989194257584" Jul 29 21:41:40.493581 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:40.493545 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="916082e2-1629-45e4-9b7a-8ac5f50078d8" path="/var/lib/kubelet/pods/916082e2-1629-45e4-9b7a-8ac5f50078d8/volumes" Jul 29 21:41:41.129712 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:41.129680 2582 generic.go:356] "Generic (PLEG): container finished" podID="f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253" containerID="7903c35d82e7dbe45696cd1f81514b53ec191d0d508e5e2e9581ddfc89d6e1a2" exitCode=0 Jul 29 21:41:41.129896 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:41.129737 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-139989194131472" event={"ID":"f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253","Type":"ContainerDied","Data":"7903c35d82e7dbe45696cd1f81514b53ec191d0d508e5e2e9581ddfc89d6e1a2"} Jul 29 21:41:42.217698 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.217659 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:41:42.277135 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.277113 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:42.305693 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.305650 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgpv6\" (UniqueName: \"kubernetes.io/projected/f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253-kube-api-access-wgpv6\") pod \"f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253\" (UID: \"f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253\") " Jul 29 21:41:42.308154 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.308129 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253-kube-api-access-wgpv6" (OuterVolumeSpecName: "kube-api-access-wgpv6") pod "f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253" (UID: "f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253"). InnerVolumeSpecName "kube-api-access-wgpv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:41:42.322667 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.322646 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7477-139989194131472_f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253/test-curl-7477-139989194131472/0.log" Jul 29 21:41:42.357994 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.357966 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7477-139989194131472"] Jul 29 21:41:42.366972 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.366946 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7477-139989194131472"] Jul 29 21:41:42.407977 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.407893 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgpv6\" (UniqueName: \"kubernetes.io/projected/f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253-kube-api-access-wgpv6\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:41:42.495124 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.495083 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253" path="/var/lib/kubelet/pods/f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253/volumes" Jul 29 21:41:42.501486 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.501444 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7477-139989194131472"] Jul 29 21:41:42.501857 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.501844 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253" containerName="test-curl-7477-139989194131472" Jul 29 21:41:42.501857 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.501860 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253" containerName="test-curl-7477-139989194131472" Jul 29 21:41:42.501962 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.501922 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f41eeb19-a9bd-4abd-a47e-ba4c6d0b6253" containerName="test-curl-7477-139989194131472" Jul 29 21:41:42.504918 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.504884 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:42.517676 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.517643 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-139989194131472"] Jul 29 21:41:42.609178 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.609144 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjpc\" (UniqueName: \"kubernetes.io/projected/4cd5bdd1-7540-4bc6-9db5-617f8221217d-kube-api-access-lkjpc\") pod \"test-curl-7477-139989194131472\" (UID: \"4cd5bdd1-7540-4bc6-9db5-617f8221217d\") " pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:42.710257 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.710160 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjpc\" (UniqueName: \"kubernetes.io/projected/4cd5bdd1-7540-4bc6-9db5-617f8221217d-kube-api-access-lkjpc\") pod \"test-curl-7477-139989194131472\" (UID: \"4cd5bdd1-7540-4bc6-9db5-617f8221217d\") " pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:42.742252 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.742148 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjpc\" (UniqueName: \"kubernetes.io/projected/4cd5bdd1-7540-4bc6-9db5-617f8221217d-kube-api-access-lkjpc\") pod \"test-curl-7477-139989194131472\" (UID: \"4cd5bdd1-7540-4bc6-9db5-617f8221217d\") " pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:42.814674 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.814638 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:42.978391 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:42.978365 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-139989194131472"] Jul 29 21:41:42.979810 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:41:42.979784 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cd5bdd1_7540_4bc6_9db5_617f8221217d.slice/crio-5f0934378568cc36c5781cf9ed828981e5fa1490bf34a9d26fa8c54bfa8736bd WatchSource:0}: Error finding container 5f0934378568cc36c5781cf9ed828981e5fa1490bf34a9d26fa8c54bfa8736bd: Status 404 returned error can't find the container with id 5f0934378568cc36c5781cf9ed828981e5fa1490bf34a9d26fa8c54bfa8736bd Jul 29 21:41:43.143864 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:43.143820 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:43.143864 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:43.143854 2582 scope.go:117] "RemoveContainer" containerID="7903c35d82e7dbe45696cd1f81514b53ec191d0d508e5e2e9581ddfc89d6e1a2" Jul 29 21:41:43.145520 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:43.145448 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-139989194131472" event={"ID":"4cd5bdd1-7540-4bc6-9db5-617f8221217d","Type":"ContainerStarted","Data":"5f0934378568cc36c5781cf9ed828981e5fa1490bf34a9d26fa8c54bfa8736bd"} Jul 29 21:41:44.154272 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:44.154234 2582 generic.go:356] "Generic (PLEG): container finished" podID="4cd5bdd1-7540-4bc6-9db5-617f8221217d" containerID="99de74421c69e3984ffb5379925f8ee6d7a699483782d2afed487b7205310a8d" exitCode=0 Jul 29 21:41:44.154272 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:44.154274 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-139989194131472" event={"ID":"4cd5bdd1-7540-4bc6-9db5-617f8221217d","Type":"ContainerDied","Data":"99de74421c69e3984ffb5379925f8ee6d7a699483782d2afed487b7205310a8d"} Jul 29 21:41:45.295956 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.295924 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:45.334999 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.334964 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkjpc\" (UniqueName: \"kubernetes.io/projected/4cd5bdd1-7540-4bc6-9db5-617f8221217d-kube-api-access-lkjpc\") pod \"4cd5bdd1-7540-4bc6-9db5-617f8221217d\" (UID: \"4cd5bdd1-7540-4bc6-9db5-617f8221217d\") " Jul 29 21:41:45.337195 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.337164 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd5bdd1-7540-4bc6-9db5-617f8221217d-kube-api-access-lkjpc" (OuterVolumeSpecName: "kube-api-access-lkjpc") pod "4cd5bdd1-7540-4bc6-9db5-617f8221217d" (UID: "4cd5bdd1-7540-4bc6-9db5-617f8221217d"). InnerVolumeSpecName "kube-api-access-lkjpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:41:45.353264 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.353235 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7477-139989194131472_4cd5bdd1-7540-4bc6-9db5-617f8221217d/test-curl-7477-139989194131472/0.log" Jul 29 21:41:45.404901 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.404865 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7477-139989194131472"] Jul 29 21:41:45.407144 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.407116 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7477-139989194131472"] Jul 29 21:41:45.436184 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.436107 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkjpc\" (UniqueName: \"kubernetes.io/projected/4cd5bdd1-7540-4bc6-9db5-617f8221217d-kube-api-access-lkjpc\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:41:45.616106 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.616071 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7477-139989194132480"] Jul 29 21:41:45.616482 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.616467 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cd5bdd1-7540-4bc6-9db5-617f8221217d" containerName="test-curl-7477-139989194131472" Jul 29 21:41:45.616527 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.616484 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cd5bdd1-7540-4bc6-9db5-617f8221217d" containerName="test-curl-7477-139989194131472" Jul 29 21:41:45.616569 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.616563 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cd5bdd1-7540-4bc6-9db5-617f8221217d" containerName="test-curl-7477-139989194131472" Jul 29 21:41:45.619642 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.619617 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194132480" Jul 29 21:41:45.649441 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.649402 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-139989194132480"] Jul 29 21:41:45.738669 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.738638 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x48wk\" (UniqueName: \"kubernetes.io/projected/06941239-7bb8-439a-919f-2292ca130911-kube-api-access-x48wk\") pod \"test-curl-7477-139989194132480\" (UID: \"06941239-7bb8-439a-919f-2292ca130911\") " pod="opendatahub/test-curl-7477-139989194132480" Jul 29 21:41:45.839763 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.839714 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x48wk\" (UniqueName: \"kubernetes.io/projected/06941239-7bb8-439a-919f-2292ca130911-kube-api-access-x48wk\") pod \"test-curl-7477-139989194132480\" (UID: \"06941239-7bb8-439a-919f-2292ca130911\") " pod="opendatahub/test-curl-7477-139989194132480" Jul 29 21:41:45.856124 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.856091 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x48wk\" (UniqueName: \"kubernetes.io/projected/06941239-7bb8-439a-919f-2292ca130911-kube-api-access-x48wk\") pod \"test-curl-7477-139989194132480\" (UID: \"06941239-7bb8-439a-919f-2292ca130911\") " pod="opendatahub/test-curl-7477-139989194132480" Jul 29 21:41:45.928891 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:45.928859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194132480" Jul 29 21:41:46.096541 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:46.096512 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-139989194132480"] Jul 29 21:41:46.098594 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:41:46.098555 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06941239_7bb8_439a_919f_2292ca130911.slice/crio-b3e3e834657425516f385c508b7d026f4e94ce86010709e6d646f7b19de3667d WatchSource:0}: Error finding container b3e3e834657425516f385c508b7d026f4e94ce86010709e6d646f7b19de3667d: Status 404 returned error can't find the container with id b3e3e834657425516f385c508b7d026f4e94ce86010709e6d646f7b19de3667d Jul 29 21:41:46.166903 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:46.166868 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-139989194132480" event={"ID":"06941239-7bb8-439a-919f-2292ca130911","Type":"ContainerStarted","Data":"b3e3e834657425516f385c508b7d026f4e94ce86010709e6d646f7b19de3667d"} Jul 29 21:41:46.168230 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:46.168210 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f0934378568cc36c5781cf9ed828981e5fa1490bf34a9d26fa8c54bfa8736bd" Jul 29 21:41:46.168230 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:46.168220 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:46.179462 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:41:46.179434 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7477-139989194131472\" is forbidden: User \"system:node:ip-10-0-137-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-160.ec2.internal' and this object" podUID="4cd5bdd1-7540-4bc6-9db5-617f8221217d" pod="opendatahub/test-curl-7477-139989194131472" Jul 29 21:41:46.493593 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:46.493557 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cd5bdd1-7540-4bc6-9db5-617f8221217d" path="/var/lib/kubelet/pods/4cd5bdd1-7540-4bc6-9db5-617f8221217d/volumes" Jul 29 21:41:47.175334 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:47.175302 2582 generic.go:356] "Generic (PLEG): container finished" podID="06941239-7bb8-439a-919f-2292ca130911" containerID="477f78c23946bc4db82fdb2a91edb954b11960a37cf7e4ab96e81a86d490627d" exitCode=0 Jul 29 21:41:47.175510 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:47.175387 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-139989194132480" event={"ID":"06941239-7bb8-439a-919f-2292ca130911","Type":"ContainerDied","Data":"477f78c23946bc4db82fdb2a91edb954b11960a37cf7e4ab96e81a86d490627d"} Jul 29 21:41:48.330107 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:48.328844 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194132480" Jul 29 21:41:48.357304 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:48.357278 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7477-139989194132480_06941239-7bb8-439a-919f-2292ca130911/test-curl-7477-139989194132480/0.log" Jul 29 21:41:48.387015 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:48.386977 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7477-139989194132480"] Jul 29 21:41:48.390630 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:48.390603 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7477-139989194132480"] Jul 29 21:41:48.467253 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:48.467153 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x48wk\" (UniqueName: \"kubernetes.io/projected/06941239-7bb8-439a-919f-2292ca130911-kube-api-access-x48wk\") pod \"06941239-7bb8-439a-919f-2292ca130911\" (UID: \"06941239-7bb8-439a-919f-2292ca130911\") " Jul 29 21:41:48.469512 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:48.469487 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06941239-7bb8-439a-919f-2292ca130911-kube-api-access-x48wk" (OuterVolumeSpecName: "kube-api-access-x48wk") pod "06941239-7bb8-439a-919f-2292ca130911" (UID: "06941239-7bb8-439a-919f-2292ca130911"). InnerVolumeSpecName "kube-api-access-x48wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:41:48.493905 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:48.493870 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06941239-7bb8-439a-919f-2292ca130911" path="/var/lib/kubelet/pods/06941239-7bb8-439a-919f-2292ca130911/volumes" Jul 29 21:41:48.568426 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:48.568388 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x48wk\" (UniqueName: \"kubernetes.io/projected/06941239-7bb8-439a-919f-2292ca130911-kube-api-access-x48wk\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:41:49.187741 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:49.187714 2582 scope.go:117] "RemoveContainer" containerID="477f78c23946bc4db82fdb2a91edb954b11960a37cf7e4ab96e81a86d490627d" Jul 29 21:41:49.187953 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:49.187714 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194132480" Jul 29 21:41:51.970465 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:51.970434 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr"] Jul 29 21:41:51.970887 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:51.970832 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06941239-7bb8-439a-919f-2292ca130911" containerName="test-curl-7477-139989194132480" Jul 29 21:41:51.970887 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:51.970844 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="06941239-7bb8-439a-919f-2292ca130911" containerName="test-curl-7477-139989194132480" Jul 29 21:41:51.970977 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:51.970901 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="06941239-7bb8-439a-919f-2292ca130911" containerName="test-curl-7477-139989194132480" Jul 29 21:41:51.975508 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:51.975492 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:51.978723 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:51.978703 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f2f2fe-openshift-default-dockercfg-b9ht2\"" Jul 29 21:41:51.999594 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:51.999552 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr"] Jul 29 21:41:52.103458 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.103421 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-podinfo\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.103458 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.103463 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-workload-certs\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.103670 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.103490 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-credential-socket\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.103670 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.103555 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-data\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.103670 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.103622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-token\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.103670 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.103650 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istiod-ca-cert\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.103794 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.103714 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-envoy\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.103794 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.103771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-workload-socket\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.103854 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.103809 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n49th\" (UniqueName: \"kubernetes.io/projected/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-kube-api-access-n49th\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.204952 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.204920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-podinfo\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.204952 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.204955 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-workload-certs\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.205169 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.204989 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-credential-socket\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.205169 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.205018 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-data\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.205169 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.205121 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-token\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.205169 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.205150 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istiod-ca-cert\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.205383 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.205186 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-envoy\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.205383 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.205236 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-workload-socket\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.205383 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.205273 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n49th\" (UniqueName: \"kubernetes.io/projected/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-kube-api-access-n49th\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.205383 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.205356 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-workload-certs\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.205599 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.205452 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-data\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.205675 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.205643 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-workload-socket\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.205736 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.205677 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-credential-socket\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.205914 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.205891 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istiod-ca-cert\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.207991 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.207965 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-envoy\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.208147 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.208129 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-podinfo\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.219954 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.219924 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n49th\" (UniqueName: \"kubernetes.io/projected/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-kube-api-access-n49th\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.220366 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.220338 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-token\") pod \"e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.285867 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.285833 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:52.456549 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.456516 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr"] Jul 29 21:41:52.461860 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:41:52.461821 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5ac923_4ed4_4ce8_8b50_ea3a65113009.slice/crio-2cf9f909007d4fb8f46bb4f637e3873d0f730225f9eb7a9432695e334655842c WatchSource:0}: Error finding container 2cf9f909007d4fb8f46bb4f637e3873d0f730225f9eb7a9432695e334655842c: Status 404 returned error can't find the container with id 2cf9f909007d4fb8f46bb4f637e3873d0f730225f9eb7a9432695e334655842c Jul 29 21:41:52.464185 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.464147 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:41:52.464304 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.464238 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:41:52.464304 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:52.464283 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:41:53.218762 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:53.218725 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" event={"ID":"6b5ac923-4ed4-4ce8-8b50-ea3a65113009","Type":"ContainerStarted","Data":"f720e4896b3d09cb8b779d203ba954992cb9a95bb80666836e0d8e933683fe43"} Jul 29 21:41:53.219302 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:53.218809 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" event={"ID":"6b5ac923-4ed4-4ce8-8b50-ea3a65113009","Type":"ContainerStarted","Data":"2cf9f909007d4fb8f46bb4f637e3873d0f730225f9eb7a9432695e334655842c"} Jul 29 21:41:53.277264 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:53.277212 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" podStartSLOduration=2.277195598 podStartE2EDuration="2.277195598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:41:53.275740961 +0000 UTC m=+3261.399330803" watchObservedRunningTime="2026-07-29 21:41:53.277195598 +0000 UTC m=+3261.400785440" Jul 29 21:41:53.286067 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:53.286013 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:53.290797 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:53.290771 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:54.224428 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:54.224393 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:41:54.225686 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:41:54.225662 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:42:03.499297 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:03.499254 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:42:10.266248 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:10.266198 2582 patch_prober.go:28] interesting pod/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 21:42:10.266710 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:10.266287 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" podUID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 21:42:12.330889 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:12.330856 2582 generic.go:356] "Generic (PLEG): container finished" podID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerID="f720e4896b3d09cb8b779d203ba954992cb9a95bb80666836e0d8e933683fe43" exitCode=0 Jul 29 21:42:12.331349 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:12.330935 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" event={"ID":"6b5ac923-4ed4-4ce8-8b50-ea3a65113009","Type":"ContainerDied","Data":"f720e4896b3d09cb8b779d203ba954992cb9a95bb80666836e0d8e933683fe43"} Jul 29 21:42:12.331394 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:12.331381 2582 scope.go:117] "RemoveContainer" containerID="f720e4896b3d09cb8b779d203ba954992cb9a95bb80666836e0d8e933683fe43" Jul 29 21:42:12.332457 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:12.332427 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:42:12.332523 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:12.332498 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:42:12.332560 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:12.332529 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:42:13.338596 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:13.338561 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" event={"ID":"6b5ac923-4ed4-4ce8-8b50-ea3a65113009","Type":"ContainerStarted","Data":"f3e2e4941bf958daf29349bfd9ca89b85a7ce1677cf57b8c5ebb1a05a9d73276"} Jul 29 21:42:14.286906 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:14.286861 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:42:14.287163 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:14.287131 2582 patch_prober.go:28] interesting pod/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Jul 29 21:42:14.287228 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:14.287187 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" podUID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Jul 29 21:42:15.286614 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:15.286575 2582 patch_prober.go:28] interesting pod/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Jul 29 21:42:15.287014 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:15.286665 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" podUID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Jul 29 21:42:16.286396 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:16.286356 2582 patch_prober.go:28] interesting pod/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Jul 29 21:42:16.286585 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:16.286426 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" podUID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Jul 29 21:42:17.290918 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:17.290890 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:42:17.291316 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:17.291275 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:42:17.292132 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:17.292112 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:42:39.175552 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.175512 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx"] Jul 29 21:42:39.179422 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.179398 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" Jul 29 21:42:39.182824 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.182799 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-00731e-dockercfg-dqdzk\"" Jul 29 21:42:39.203293 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.203257 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx"] Jul 29 21:42:39.208817 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.208788 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-00731e\"" Jul 29 21:42:39.284616 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.284576 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcmzg\" (UniqueName: \"kubernetes.io/projected/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e-kube-api-access-dcmzg\") pod \"payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx\" (UID: \"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" Jul 29 21:42:39.284803 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.284719 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx\" (UID: \"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" Jul 29 21:42:39.385439 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.385400 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcmzg\" (UniqueName: \"kubernetes.io/projected/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e-kube-api-access-dcmzg\") pod \"payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx\" (UID: \"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" Jul 29 21:42:39.385702 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.385494 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx\" (UID: \"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" Jul 29 21:42:39.386178 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.386161 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx\" (UID: \"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" Jul 29 21:42:39.413283 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.413246 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcmzg\" (UniqueName: \"kubernetes.io/projected/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e-kube-api-access-dcmzg\") pod \"payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx\" (UID: \"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" Jul 29 21:42:39.489739 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.489711 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" Jul 29 21:42:39.672058 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:42:39.671978 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ecafe6_6c15_4d90_9dc3_23fe50f55b0e.slice/crio-ddce29056ebc47daa8dfd1e307c5d761610843a34574201840bc4a22a8f9d05d WatchSource:0}: Error finding container ddce29056ebc47daa8dfd1e307c5d761610843a34574201840bc4a22a8f9d05d: Status 404 returned error can't find the container with id ddce29056ebc47daa8dfd1e307c5d761610843a34574201840bc4a22a8f9d05d Jul 29 21:42:39.673592 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:39.673566 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx"] Jul 29 21:42:40.494763 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:40.494725 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" event={"ID":"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e","Type":"ContainerStarted","Data":"38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d"} Jul 29 21:42:40.495157 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:40.494783 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" event={"ID":"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e","Type":"ContainerStarted","Data":"ddce29056ebc47daa8dfd1e307c5d761610843a34574201840bc4a22a8f9d05d"} Jul 29 21:42:40.495157 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:40.494824 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" Jul 29 21:42:40.527446 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:40.527395 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" podStartSLOduration=1.527378173 podStartE2EDuration="1.527378173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:42:40.525525544 +0000 UTC m=+3308.649115385" watchObservedRunningTime="2026-07-29 21:42:40.527378173 +0000 UTC m=+3308.650968014" Jul 29 21:42:42.526415 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:42.526377 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:42:51.503908 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:42:51.503873 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" Jul 29 21:43:02.172393 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:43:02.172259 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx_b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e/payload-processing/0.log" Jul 29 21:43:26.464932 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:43:26.464886 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:43:54.256827 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:43:54.256793 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:44:04.756246 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:04.756218 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx_b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e/payload-processing/0.log" Jul 29 21:44:12.457222 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:12.457184 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k"] Jul 29 21:44:12.460969 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:12.460946 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" Jul 29 21:44:12.463564 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:12.463543 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-50783d-dockercfg-m9df8\"" Jul 29 21:44:12.463696 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:12.463586 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-50783d\"" Jul 29 21:44:12.472482 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:12.472460 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k"] Jul 29 21:44:12.504563 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:12.504530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k\" (UID: \"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" Jul 29 21:44:12.504731 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:12.504592 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grnj\" (UniqueName: \"kubernetes.io/projected/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3-kube-api-access-4grnj\") pod \"payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k\" (UID: \"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" Jul 29 21:44:12.605430 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:12.605392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k\" (UID: \"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" Jul 29 21:44:12.605609 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:12.605454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4grnj\" (UniqueName: \"kubernetes.io/projected/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3-kube-api-access-4grnj\") pod \"payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k\" (UID: \"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" Jul 29 21:44:12.606073 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:12.606055 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k\" (UID: \"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" Jul 29 21:44:12.614723 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:12.614704 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grnj\" (UniqueName: \"kubernetes.io/projected/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3-kube-api-access-4grnj\") pod \"payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k\" (UID: \"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" Jul 29 21:44:12.772721 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:12.772690 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" Jul 29 21:44:13.126187 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:13.126162 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k"] Jul 29 21:44:13.128102 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:44:13.128070 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d5fb12_9a1e_4c4f_9add_9bd0576105c3.slice/crio-bc76b04de18ef4cd96e86dacc2a99c1825edc71ae0b9c19c6e9e60977d25ff27 WatchSource:0}: Error finding container bc76b04de18ef4cd96e86dacc2a99c1825edc71ae0b9c19c6e9e60977d25ff27: Status 404 returned error can't find the container with id bc76b04de18ef4cd96e86dacc2a99c1825edc71ae0b9c19c6e9e60977d25ff27 Jul 29 21:44:14.021638 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:14.021600 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" event={"ID":"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3","Type":"ContainerStarted","Data":"03998f25598ad12291e780dbc1f1ac552286b8a438bc9e6abd23c6e28a9c04b6"} Jul 29 21:44:14.022062 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:14.021671 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" Jul 29 21:44:14.022062 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:14.021692 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" event={"ID":"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3","Type":"ContainerStarted","Data":"bc76b04de18ef4cd96e86dacc2a99c1825edc71ae0b9c19c6e9e60977d25ff27"} Jul 29 21:44:14.040417 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:14.040372 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" podStartSLOduration=2.040355104 podStartE2EDuration="2.040355104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:44:14.03852838 +0000 UTC m=+3402.162118222" watchObservedRunningTime="2026-07-29 21:44:14.040355104 +0000 UTC m=+3402.163944991" Jul 29 21:44:20.660499 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:20.660459 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:44:25.029922 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:25.029840 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" Jul 29 21:44:37.992900 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:37.992861 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k"] Jul 29 21:44:37.993556 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:37.993158 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" podUID="a3d5fb12-9a1e-4c4f-9add-9bd0576105c3" containerName="payload-processing" containerID="cri-o://03998f25598ad12291e780dbc1f1ac552286b8a438bc9e6abd23c6e28a9c04b6" gracePeriod=30 Jul 29 21:44:38.161337 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:38.161290 2582 generic.go:356] "Generic (PLEG): container finished" podID="a3d5fb12-9a1e-4c4f-9add-9bd0576105c3" containerID="03998f25598ad12291e780dbc1f1ac552286b8a438bc9e6abd23c6e28a9c04b6" exitCode=0 Jul 29 21:44:38.161696 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:38.161619 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" event={"ID":"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3","Type":"ContainerDied","Data":"03998f25598ad12291e780dbc1f1ac552286b8a438bc9e6abd23c6e28a9c04b6"} Jul 29 21:44:38.273279 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:38.273254 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" Jul 29 21:44:38.361851 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:38.361819 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4grnj\" (UniqueName: \"kubernetes.io/projected/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3-kube-api-access-4grnj\") pod \"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3\" (UID: \"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3\") " Jul 29 21:44:38.362071 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:38.361945 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3-plugins-config-volume\") pod \"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3\" (UID: \"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3\") " Jul 29 21:44:38.362317 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:38.362293 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3d5fb12-9a1e-4c4f-9add-9bd0576105c3" (UID: "a3d5fb12-9a1e-4c4f-9add-9bd0576105c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:44:38.364190 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:38.364165 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3-kube-api-access-4grnj" (OuterVolumeSpecName: "kube-api-access-4grnj") pod "a3d5fb12-9a1e-4c4f-9add-9bd0576105c3" (UID: "a3d5fb12-9a1e-4c4f-9add-9bd0576105c3"). InnerVolumeSpecName "kube-api-access-4grnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:44:38.463510 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:38.463472 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4grnj\" (UniqueName: \"kubernetes.io/projected/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3-kube-api-access-4grnj\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:44:38.463510 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:38.463502 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:44:39.167643 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:39.167611 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" Jul 29 21:44:39.167643 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:39.167625 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k" event={"ID":"a3d5fb12-9a1e-4c4f-9add-9bd0576105c3","Type":"ContainerDied","Data":"bc76b04de18ef4cd96e86dacc2a99c1825edc71ae0b9c19c6e9e60977d25ff27"} Jul 29 21:44:39.168233 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:39.167689 2582 scope.go:117] "RemoveContainer" containerID="03998f25598ad12291e780dbc1f1ac552286b8a438bc9e6abd23c6e28a9c04b6" Jul 29 21:44:39.185810 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:39.185786 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k"] Jul 29 21:44:39.193233 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:39.193208 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-50783d-cf56d55c8-bdc9k"] Jul 29 21:44:40.493300 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:40.493265 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3d5fb12-9a1e-4c4f-9add-9bd0576105c3" path="/var/lib/kubelet/pods/a3d5fb12-9a1e-4c4f-9add-9bd0576105c3/volumes" Jul 29 21:44:42.297046 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:44:42.296991 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:45:00.272520 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:00.272481 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:45:11.559934 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:11.559889 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:45:51.459215 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:51.459180 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr"] Jul 29 21:45:51.459804 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:51.459427 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" podUID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerName="istio-proxy" containerID="cri-o://f3e2e4941bf958daf29349bfd9ca89b85a7ce1677cf57b8c5ebb1a05a9d73276" gracePeriod=30 Jul 29 21:45:56.617059 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.617001 2582 generic.go:356] "Generic (PLEG): container finished" podID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerID="f3e2e4941bf958daf29349bfd9ca89b85a7ce1677cf57b8c5ebb1a05a9d73276" exitCode=0 Jul 29 21:45:56.617497 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.617065 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" event={"ID":"6b5ac923-4ed4-4ce8-8b50-ea3a65113009","Type":"ContainerDied","Data":"f3e2e4941bf958daf29349bfd9ca89b85a7ce1677cf57b8c5ebb1a05a9d73276"} Jul 29 21:45:56.617497 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.617140 2582 scope.go:117] "RemoveContainer" containerID="f720e4896b3d09cb8b779d203ba954992cb9a95bb80666836e0d8e933683fe43" Jul 29 21:45:56.722970 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.722905 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:45:56.749781 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.749744 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istiod-ca-cert\") pod \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " Jul 29 21:45:56.749967 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.749796 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-envoy\") pod \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " Jul 29 21:45:56.749967 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.749836 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-podinfo\") pod \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " Jul 29 21:45:56.749967 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.749881 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n49th\" (UniqueName: \"kubernetes.io/projected/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-kube-api-access-n49th\") pod \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " Jul 29 21:45:56.749967 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.749947 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-data\") pod \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " Jul 29 21:45:56.750262 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.749974 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-workload-socket\") pod \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " Jul 29 21:45:56.750262 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.750075 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-token\") pod \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " Jul 29 21:45:56.750262 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.750127 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-credential-socket\") pod \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " Jul 29 21:45:56.750262 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.750147 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b5ac923-4ed4-4ce8-8b50-ea3a65113009" (UID: "6b5ac923-4ed4-4ce8-8b50-ea3a65113009"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:45:56.750262 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.750158 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-workload-certs\") pod \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\" (UID: \"6b5ac923-4ed4-4ce8-8b50-ea3a65113009\") " Jul 29 21:45:56.750536 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.750372 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b5ac923-4ed4-4ce8-8b50-ea3a65113009" (UID: "6b5ac923-4ed4-4ce8-8b50-ea3a65113009"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:45:56.750596 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.750496 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b5ac923-4ed4-4ce8-8b50-ea3a65113009" (UID: "6b5ac923-4ed4-4ce8-8b50-ea3a65113009"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:45:56.751749 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.750805 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:45:56.751749 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.750830 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:45:56.751749 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.750844 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:45:56.751749 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.750971 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b5ac923-4ed4-4ce8-8b50-ea3a65113009" (UID: "6b5ac923-4ed4-4ce8-8b50-ea3a65113009"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:45:56.751749 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.751710 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b5ac923-4ed4-4ce8-8b50-ea3a65113009" (UID: "6b5ac923-4ed4-4ce8-8b50-ea3a65113009"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:45:56.752527 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.752500 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-kube-api-access-n49th" (OuterVolumeSpecName: "kube-api-access-n49th") pod "6b5ac923-4ed4-4ce8-8b50-ea3a65113009" (UID: "6b5ac923-4ed4-4ce8-8b50-ea3a65113009"). InnerVolumeSpecName "kube-api-access-n49th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:45:56.752744 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.752717 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b5ac923-4ed4-4ce8-8b50-ea3a65113009" (UID: "6b5ac923-4ed4-4ce8-8b50-ea3a65113009"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:45:56.752943 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.752920 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b5ac923-4ed4-4ce8-8b50-ea3a65113009" (UID: "6b5ac923-4ed4-4ce8-8b50-ea3a65113009"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:45:56.753105 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.752987 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b5ac923-4ed4-4ce8-8b50-ea3a65113009" (UID: "6b5ac923-4ed4-4ce8-8b50-ea3a65113009"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:45:56.852080 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.851991 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:45:56.852080 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.852065 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:45:56.852080 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.852077 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:45:56.852080 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.852085 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n49th\" (UniqueName: \"kubernetes.io/projected/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-kube-api-access-n49th\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:45:56.852080 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.852094 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:45:56.852402 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:56.852104 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b5ac923-4ed4-4ce8-8b50-ea3a65113009-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:45:57.624685 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:57.624645 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" event={"ID":"6b5ac923-4ed4-4ce8-8b50-ea3a65113009","Type":"ContainerDied","Data":"2cf9f909007d4fb8f46bb4f637e3873d0f730225f9eb7a9432695e334655842c"} Jul 29 21:45:57.624685 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:57.624678 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr" Jul 29 21:45:57.625319 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:57.624716 2582 scope.go:117] "RemoveContainer" containerID="f3e2e4941bf958daf29349bfd9ca89b85a7ce1677cf57b8c5ebb1a05a9d73276" Jul 29 21:45:57.649761 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:57.649728 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr"] Jul 29 21:45:57.662898 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:57.662868 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f2f2fe-openshift-default-cf68b8f8d-rjqbr"] Jul 29 21:45:58.493119 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:58.493081 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" path="/var/lib/kubelet/pods/6b5ac923-4ed4-4ce8-8b50-ea3a65113009/volumes" Jul 29 21:45:58.631375 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:58.631345 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx"] Jul 29 21:45:58.631819 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:58.631589 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" podUID="b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e" containerName="payload-processing" containerID="cri-o://38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d" gracePeriod=30 Jul 29 21:45:58.906953 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:58.906924 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" Jul 29 21:45:58.973359 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:58.973321 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e-plugins-config-volume\") pod \"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e\" (UID: \"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e\") " Jul 29 21:45:58.973541 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:58.973385 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcmzg\" (UniqueName: \"kubernetes.io/projected/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e-kube-api-access-dcmzg\") pod \"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e\" (UID: \"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e\") " Jul 29 21:45:58.973729 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:58.973705 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e" (UID: "b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:45:58.975682 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:58.975648 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e-kube-api-access-dcmzg" (OuterVolumeSpecName: "kube-api-access-dcmzg") pod "b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e" (UID: "b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e"). InnerVolumeSpecName "kube-api-access-dcmzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:45:59.075178 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:59.075081 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:45:59.075178 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:59.075115 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcmzg\" (UniqueName: \"kubernetes.io/projected/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e-kube-api-access-dcmzg\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:45:59.639149 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:59.639115 2582 generic.go:356] "Generic (PLEG): container finished" podID="b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e" containerID="38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d" exitCode=0 Jul 29 21:45:59.639667 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:59.639166 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" Jul 29 21:45:59.639667 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:59.639212 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" event={"ID":"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e","Type":"ContainerDied","Data":"38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d"} Jul 29 21:45:59.639667 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:59.639267 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx" event={"ID":"b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e","Type":"ContainerDied","Data":"ddce29056ebc47daa8dfd1e307c5d761610843a34574201840bc4a22a8f9d05d"} Jul 29 21:45:59.639667 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:59.639295 2582 scope.go:117] "RemoveContainer" containerID="38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d" Jul 29 21:45:59.649053 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:59.649009 2582 scope.go:117] "RemoveContainer" containerID="38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d" Jul 29 21:45:59.649310 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:45:59.649291 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d\": container with ID starting with 38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d not found: ID does not exist" containerID="38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d" Jul 29 21:45:59.649377 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:59.649320 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d"} err="failed to get container status \"38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d\": rpc error: code = NotFound desc = could not find container \"38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d\": container with ID starting with 38d108d0b394ac362b84b1d9c879baae39aa25017a3298ba61843f9391b26d2d not found: ID does not exist" Jul 29 21:45:59.663091 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:59.663062 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx"] Jul 29 21:45:59.670017 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:45:59.669991 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00731e-67978cf968-fxfbx"] Jul 29 21:46:00.493613 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:00.493579 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e" path="/var/lib/kubelet/pods/b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e/volumes" Jul 29 21:46:01.658877 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:01.658841 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:46:06.438626 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.438587 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns"] Jul 29 21:46:06.439057 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.438968 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e" containerName="payload-processing" Jul 29 21:46:06.439057 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.438979 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e" containerName="payload-processing" Jul 29 21:46:06.439057 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.438988 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d5fb12-9a1e-4c4f-9add-9bd0576105c3" containerName="payload-processing" Jul 29 21:46:06.439057 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.438993 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d5fb12-9a1e-4c4f-9add-9bd0576105c3" containerName="payload-processing" Jul 29 21:46:06.439057 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.439011 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerName="istio-proxy" Jul 29 21:46:06.439057 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.439016 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerName="istio-proxy" Jul 29 21:46:06.439057 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.439043 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerName="istio-proxy" Jul 29 21:46:06.439057 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.439048 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerName="istio-proxy" Jul 29 21:46:06.439380 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.439109 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerName="istio-proxy" Jul 29 21:46:06.439380 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.439116 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ecafe6-6c15-4d90-9dc3-23fe50f55b0e" containerName="payload-processing" Jul 29 21:46:06.439380 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.439122 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b5ac923-4ed4-4ce8-8b50-ea3a65113009" containerName="istio-proxy" Jul 29 21:46:06.439380 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.439129 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d5fb12-9a1e-4c4f-9add-9bd0576105c3" containerName="payload-processing" Jul 29 21:46:06.443663 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.443643 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns" Jul 29 21:46:06.447902 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.447871 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:46:06.448097 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.447918 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\"" Jul 29 21:46:06.452333 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.452306 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns"] Jul 29 21:46:06.548180 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.548144 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d631c8d8-d099-4864-a860-c60ec038f5ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns\" (UID: \"d631c8d8-d099-4864-a860-c60ec038f5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns" Jul 29 21:46:06.649782 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.649737 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d631c8d8-d099-4864-a860-c60ec038f5ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns\" (UID: \"d631c8d8-d099-4864-a860-c60ec038f5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns" Jul 29 21:46:06.650409 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.650390 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d631c8d8-d099-4864-a860-c60ec038f5ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns\" (UID: \"d631c8d8-d099-4864-a860-c60ec038f5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns" Jul 29 21:46:06.755515 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.755484 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns" Jul 29 21:46:06.897967 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:06.897937 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns"] Jul 29 21:46:06.900211 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:46:06.900181 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd631c8d8_d099_4864_a860_c60ec038f5ef.slice/crio-f3576ba6417556c5d3ddf1a9f2527b0b8fb99ddcd84a64d5efc33fb27389ccae WatchSource:0}: Error finding container f3576ba6417556c5d3ddf1a9f2527b0b8fb99ddcd84a64d5efc33fb27389ccae: Status 404 returned error can't find the container with id f3576ba6417556c5d3ddf1a9f2527b0b8fb99ddcd84a64d5efc33fb27389ccae Jul 29 21:46:07.686472 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:07.686436 2582 generic.go:356] "Generic (PLEG): container finished" podID="d631c8d8-d099-4864-a860-c60ec038f5ef" containerID="c5bdc2db5628a654d2ec4f6da53ccf6e88f016d90c24cf8ff90e2a222394a5c1" exitCode=0 Jul 29 21:46:07.686883 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:07.686518 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns" event={"ID":"d631c8d8-d099-4864-a860-c60ec038f5ef","Type":"ContainerDied","Data":"c5bdc2db5628a654d2ec4f6da53ccf6e88f016d90c24cf8ff90e2a222394a5c1"} Jul 29 21:46:07.686883 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:07.686571 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns" event={"ID":"d631c8d8-d099-4864-a860-c60ec038f5ef","Type":"ContainerStarted","Data":"f3576ba6417556c5d3ddf1a9f2527b0b8fb99ddcd84a64d5efc33fb27389ccae"} Jul 29 21:46:08.833261 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:08.833232 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns" Jul 29 21:46:08.976347 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:08.976255 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d631c8d8-d099-4864-a860-c60ec038f5ef-maas-api-service-ca\") pod \"d631c8d8-d099-4864-a860-c60ec038f5ef\" (UID: \"d631c8d8-d099-4864-a860-c60ec038f5ef\") " Jul 29 21:46:08.976623 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:08.976599 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d631c8d8-d099-4864-a860-c60ec038f5ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d631c8d8-d099-4864-a860-c60ec038f5ef" (UID: "d631c8d8-d099-4864-a860-c60ec038f5ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:46:09.077671 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:09.077632 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d631c8d8-d099-4864-a860-c60ec038f5ef-maas-api-service-ca\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:09.698699 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:09.698668 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns" Jul 29 21:46:09.698699 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:09.698672 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns" event={"ID":"d631c8d8-d099-4864-a860-c60ec038f5ef","Type":"ContainerDied","Data":"f3576ba6417556c5d3ddf1a9f2527b0b8fb99ddcd84a64d5efc33fb27389ccae"} Jul 29 21:46:09.698922 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:09.698764 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3576ba6417556c5d3ddf1a9f2527b0b8fb99ddcd84a64d5efc33fb27389ccae" Jul 29 21:46:12.253612 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.253578 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9"] Jul 29 21:46:12.254107 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.253804 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" podUID="6bcaed3c-f027-4b54-9ea7-cda91977fd92" containerName="payload-processing" containerID="cri-o://6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0" gracePeriod=30 Jul 29 21:46:12.259061 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.259007 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd"] Jul 29 21:46:12.259287 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.259255 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" podUID="b7bb0446-62f0-435d-bc3e-580f02c6cf8f" containerName="payload-pre-processing" containerID="cri-o://dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848" gracePeriod=30 Jul 29 21:46:12.411885 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.411848 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns"] Jul 29 21:46:12.417884 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.417847 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eb3087-rxtns"] Jul 29 21:46:12.495992 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.495955 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d631c8d8-d099-4864-a860-c60ec038f5ef" path="/var/lib/kubelet/pods/d631c8d8-d099-4864-a860-c60ec038f5ef/volumes" Jul 29 21:46:12.533340 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.533316 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" Jul 29 21:46:12.575453 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.575429 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" Jul 29 21:46:12.612733 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.612659 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nxqb\" (UniqueName: \"kubernetes.io/projected/6bcaed3c-f027-4b54-9ea7-cda91977fd92-kube-api-access-8nxqb\") pod \"6bcaed3c-f027-4b54-9ea7-cda91977fd92\" (UID: \"6bcaed3c-f027-4b54-9ea7-cda91977fd92\") " Jul 29 21:46:12.612733 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.612726 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcaed3c-f027-4b54-9ea7-cda91977fd92-plugins-config-volume\") pod \"6bcaed3c-f027-4b54-9ea7-cda91977fd92\" (UID: \"6bcaed3c-f027-4b54-9ea7-cda91977fd92\") " Jul 29 21:46:12.613268 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.613231 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcaed3c-f027-4b54-9ea7-cda91977fd92-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6bcaed3c-f027-4b54-9ea7-cda91977fd92" (UID: "6bcaed3c-f027-4b54-9ea7-cda91977fd92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:46:12.615066 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.615015 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcaed3c-f027-4b54-9ea7-cda91977fd92-kube-api-access-8nxqb" (OuterVolumeSpecName: "kube-api-access-8nxqb") pod "6bcaed3c-f027-4b54-9ea7-cda91977fd92" (UID: "6bcaed3c-f027-4b54-9ea7-cda91977fd92"). InnerVolumeSpecName "kube-api-access-8nxqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:46:12.713778 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.713749 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bb0446-62f0-435d-bc3e-580f02c6cf8f-plugins-config-volume\") pod \"b7bb0446-62f0-435d-bc3e-580f02c6cf8f\" (UID: \"b7bb0446-62f0-435d-bc3e-580f02c6cf8f\") " Jul 29 21:46:12.713778 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.713786 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtcwk\" (UniqueName: \"kubernetes.io/projected/b7bb0446-62f0-435d-bc3e-580f02c6cf8f-kube-api-access-gtcwk\") pod \"b7bb0446-62f0-435d-bc3e-580f02c6cf8f\" (UID: \"b7bb0446-62f0-435d-bc3e-580f02c6cf8f\") " Jul 29 21:46:12.714092 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.714077 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nxqb\" (UniqueName: \"kubernetes.io/projected/6bcaed3c-f027-4b54-9ea7-cda91977fd92-kube-api-access-8nxqb\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:12.714151 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.714094 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcaed3c-f027-4b54-9ea7-cda91977fd92-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:12.714259 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.714231 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7bb0446-62f0-435d-bc3e-580f02c6cf8f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7bb0446-62f0-435d-bc3e-580f02c6cf8f" (UID: "b7bb0446-62f0-435d-bc3e-580f02c6cf8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:46:12.716168 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.716137 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bb0446-62f0-435d-bc3e-580f02c6cf8f-kube-api-access-gtcwk" (OuterVolumeSpecName: "kube-api-access-gtcwk") pod "b7bb0446-62f0-435d-bc3e-580f02c6cf8f" (UID: "b7bb0446-62f0-435d-bc3e-580f02c6cf8f"). InnerVolumeSpecName "kube-api-access-gtcwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:46:12.717798 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.717770 2582 generic.go:356] "Generic (PLEG): container finished" podID="b7bb0446-62f0-435d-bc3e-580f02c6cf8f" containerID="dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848" exitCode=0 Jul 29 21:46:12.717912 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.717816 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" Jul 29 21:46:12.717912 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.717857 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" event={"ID":"b7bb0446-62f0-435d-bc3e-580f02c6cf8f","Type":"ContainerDied","Data":"dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848"} Jul 29 21:46:12.717991 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.717914 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd" event={"ID":"b7bb0446-62f0-435d-bc3e-580f02c6cf8f","Type":"ContainerDied","Data":"f21b8142396bf909d182a458170767a4aae1783b30d1ca5ec9db924584f3e2e5"} Jul 29 21:46:12.717991 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.717945 2582 scope.go:117] "RemoveContainer" containerID="dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848" Jul 29 21:46:12.719557 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.719532 2582 generic.go:356] "Generic (PLEG): container finished" podID="6bcaed3c-f027-4b54-9ea7-cda91977fd92" containerID="6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0" exitCode=0 Jul 29 21:46:12.719658 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.719576 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" Jul 29 21:46:12.719716 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.719575 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" event={"ID":"6bcaed3c-f027-4b54-9ea7-cda91977fd92","Type":"ContainerDied","Data":"6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0"} Jul 29 21:46:12.719716 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.719704 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9" event={"ID":"6bcaed3c-f027-4b54-9ea7-cda91977fd92","Type":"ContainerDied","Data":"b081e04a9f7160d480d236f19bd015a5a8ad371414d96e74efc8e040e2169c6f"} Jul 29 21:46:12.728834 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.728813 2582 scope.go:117] "RemoveContainer" containerID="dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848" Jul 29 21:46:12.729273 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:46:12.729241 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848\": container with ID starting with dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848 not found: ID does not exist" containerID="dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848" Jul 29 21:46:12.729390 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.729285 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848"} err="failed to get container status \"dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848\": rpc error: code = NotFound desc = could not find container \"dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848\": container with ID starting with dbbbfb736c90021d840332e3a50323cd4e301d13e3eca4ef8201de4ae0975848 not found: ID does not exist" Jul 29 21:46:12.729390 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.729313 2582 scope.go:117] "RemoveContainer" containerID="6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0" Jul 29 21:46:12.738340 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.738320 2582 scope.go:117] "RemoveContainer" containerID="6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0" Jul 29 21:46:12.738649 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:46:12.738628 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0\": container with ID starting with 6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0 not found: ID does not exist" containerID="6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0" Jul 29 21:46:12.738787 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.738655 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0"} err="failed to get container status \"6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0\": rpc error: code = NotFound desc = could not find container \"6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0\": container with ID starting with 6cd17e7d58a15568e3f53165fa9b7fa25a10def37ba286948ca030e5d58c48d0 not found: ID does not exist" Jul 29 21:46:12.744784 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.744761 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd"] Jul 29 21:46:12.746936 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.746917 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-eb3087-6787769688-fgpsd"] Jul 29 21:46:12.759805 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.759733 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9"] Jul 29 21:46:12.766146 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.766122 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-eb3087-8855c856b-m49p9"] Jul 29 21:46:12.815387 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.815347 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bb0446-62f0-435d-bc3e-580f02c6cf8f-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:12.815387 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:12.815379 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtcwk\" (UniqueName: \"kubernetes.io/projected/b7bb0446-62f0-435d-bc3e-580f02c6cf8f-kube-api-access-gtcwk\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:14.493002 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:14.492967 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bcaed3c-f027-4b54-9ea7-cda91977fd92" path="/var/lib/kubelet/pods/6bcaed3c-f027-4b54-9ea7-cda91977fd92/volumes" Jul 29 21:46:14.493456 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:14.493399 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7bb0446-62f0-435d-bc3e-580f02c6cf8f" path="/var/lib/kubelet/pods/b7bb0446-62f0-435d-bc3e-580f02c6cf8f/volumes" Jul 29 21:46:15.855113 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:15.855077 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:46:19.856202 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:19.856170 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms"] Jul 29 21:46:19.856647 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:19.856392 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" containerID="cri-o://8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb" gracePeriod=30 Jul 29 21:46:25.132696 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.132672 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:46:25.235060 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235014 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-istio-envoy\") pod \"171461e2-b759-4116-992b-14d0feb8005f\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " Jul 29 21:46:25.235060 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235059 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171461e2-b759-4116-992b-14d0feb8005f-istiod-ca-cert\") pod \"171461e2-b759-4116-992b-14d0feb8005f\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " Jul 29 21:46:25.235292 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235094 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171461e2-b759-4116-992b-14d0feb8005f-istio-podinfo\") pod \"171461e2-b759-4116-992b-14d0feb8005f\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " Jul 29 21:46:25.235292 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235113 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-istio-data\") pod \"171461e2-b759-4116-992b-14d0feb8005f\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " Jul 29 21:46:25.235292 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235138 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-workload-socket\") pod \"171461e2-b759-4116-992b-14d0feb8005f\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " Jul 29 21:46:25.235292 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235162 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmwsb\" (UniqueName: \"kubernetes.io/projected/171461e2-b759-4116-992b-14d0feb8005f-kube-api-access-xmwsb\") pod \"171461e2-b759-4116-992b-14d0feb8005f\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " Jul 29 21:46:25.235292 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235250 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-credential-socket\") pod \"171461e2-b759-4116-992b-14d0feb8005f\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " Jul 29 21:46:25.235292 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235279 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171461e2-b759-4116-992b-14d0feb8005f-istio-token\") pod \"171461e2-b759-4116-992b-14d0feb8005f\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " Jul 29 21:46:25.235602 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235333 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-workload-certs\") pod \"171461e2-b759-4116-992b-14d0feb8005f\" (UID: \"171461e2-b759-4116-992b-14d0feb8005f\") " Jul 29 21:46:25.235602 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235472 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "171461e2-b759-4116-992b-14d0feb8005f" (UID: "171461e2-b759-4116-992b-14d0feb8005f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:46:25.235722 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235604 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "171461e2-b759-4116-992b-14d0feb8005f" (UID: "171461e2-b759-4116-992b-14d0feb8005f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:46:25.235722 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235660 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:25.235722 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235702 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171461e2-b759-4116-992b-14d0feb8005f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "171461e2-b759-4116-992b-14d0feb8005f" (UID: "171461e2-b759-4116-992b-14d0feb8005f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:46:25.235881 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.235812 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "171461e2-b759-4116-992b-14d0feb8005f" (UID: "171461e2-b759-4116-992b-14d0feb8005f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:46:25.236071 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.236052 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-istio-data" (OuterVolumeSpecName: "istio-data") pod "171461e2-b759-4116-992b-14d0feb8005f" (UID: "171461e2-b759-4116-992b-14d0feb8005f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:46:25.237464 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.237436 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "171461e2-b759-4116-992b-14d0feb8005f" (UID: "171461e2-b759-4116-992b-14d0feb8005f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:46:25.237596 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.237548 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/171461e2-b759-4116-992b-14d0feb8005f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "171461e2-b759-4116-992b-14d0feb8005f" (UID: "171461e2-b759-4116-992b-14d0feb8005f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:46:25.237802 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.237782 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171461e2-b759-4116-992b-14d0feb8005f-istio-token" (OuterVolumeSpecName: "istio-token") pod "171461e2-b759-4116-992b-14d0feb8005f" (UID: "171461e2-b759-4116-992b-14d0feb8005f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:46:25.237939 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.237919 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171461e2-b759-4116-992b-14d0feb8005f-kube-api-access-xmwsb" (OuterVolumeSpecName: "kube-api-access-xmwsb") pod "171461e2-b759-4116-992b-14d0feb8005f" (UID: "171461e2-b759-4116-992b-14d0feb8005f"). InnerVolumeSpecName "kube-api-access-xmwsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:46:25.337161 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.337123 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171461e2-b759-4116-992b-14d0feb8005f-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:25.337161 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.337158 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:25.337161 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.337168 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmwsb\" (UniqueName: \"kubernetes.io/projected/171461e2-b759-4116-992b-14d0feb8005f-kube-api-access-xmwsb\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:25.337417 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.337178 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:25.337417 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.337187 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171461e2-b759-4116-992b-14d0feb8005f-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:25.337417 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.337196 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:25.337417 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.337203 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171461e2-b759-4116-992b-14d0feb8005f-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:25.337417 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.337210 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171461e2-b759-4116-992b-14d0feb8005f-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:25.585592 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.585554 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5"] Jul 29 21:46:25.585861 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.585814 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" podUID="5a10363e-a2d1-437f-81c8-842810aad721" containerName="payload-processing" containerID="cri-o://71ae872f417b24ed27ac1183dbac146a0089d89df21d3280e6714e756a9b1c2e" gracePeriod=30 Jul 29 21:46:25.598519 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.597104 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52"] Jul 29 21:46:25.598519 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.597341 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" podUID="c345ed65-4a4f-4b61-ac8e-cf0daa593f6b" containerName="payload-pre-processing" containerID="cri-o://5f112522c05bf6e85b465f003a5c7aa77648bbd1a5b4dc6097ea64907b13dcf6" gracePeriod=30 Jul 29 21:46:25.804826 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.803687 2582 generic.go:356] "Generic (PLEG): container finished" podID="171461e2-b759-4116-992b-14d0feb8005f" containerID="8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb" exitCode=0 Jul 29 21:46:25.804826 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.803780 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" Jul 29 21:46:25.804826 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.803913 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" event={"ID":"171461e2-b759-4116-992b-14d0feb8005f","Type":"ContainerDied","Data":"8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb"} Jul 29 21:46:25.804826 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.803990 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms" event={"ID":"171461e2-b759-4116-992b-14d0feb8005f","Type":"ContainerDied","Data":"793f39a49f542120cc63849b1a7e218b98d0d19774c09ffa88140a9eea68dcbc"} Jul 29 21:46:25.804826 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.804050 2582 scope.go:117] "RemoveContainer" containerID="8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb" Jul 29 21:46:25.806434 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.806335 2582 generic.go:356] "Generic (PLEG): container finished" podID="c345ed65-4a4f-4b61-ac8e-cf0daa593f6b" containerID="5f112522c05bf6e85b465f003a5c7aa77648bbd1a5b4dc6097ea64907b13dcf6" exitCode=0 Jul 29 21:46:25.806434 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.806410 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" event={"ID":"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b","Type":"ContainerDied","Data":"5f112522c05bf6e85b465f003a5c7aa77648bbd1a5b4dc6097ea64907b13dcf6"} Jul 29 21:46:25.809742 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.809587 2582 generic.go:356] "Generic (PLEG): container finished" podID="5a10363e-a2d1-437f-81c8-842810aad721" containerID="71ae872f417b24ed27ac1183dbac146a0089d89df21d3280e6714e756a9b1c2e" exitCode=0 Jul 29 21:46:25.809742 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.809681 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" event={"ID":"5a10363e-a2d1-437f-81c8-842810aad721","Type":"ContainerDied","Data":"71ae872f417b24ed27ac1183dbac146a0089d89df21d3280e6714e756a9b1c2e"} Jul 29 21:46:25.818864 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.818843 2582 scope.go:117] "RemoveContainer" containerID="10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308" Jul 29 21:46:25.834137 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.834102 2582 scope.go:117] "RemoveContainer" containerID="8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb" Jul 29 21:46:25.834544 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:46:25.834509 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb\": container with ID starting with 8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb not found: ID does not exist" containerID="8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb" Jul 29 21:46:25.834671 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.834544 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb"} err="failed to get container status \"8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb\": rpc error: code = NotFound desc = could not find container \"8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb\": container with ID starting with 8dc10241157bdf9db2d18a389803145b0e2b8784813ba6a578247374ff9d7abb not found: ID does not exist" Jul 29 21:46:25.834671 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.834564 2582 scope.go:117] "RemoveContainer" containerID="10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308" Jul 29 21:46:25.834960 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:46:25.834903 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308\": container with ID starting with 10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308 not found: ID does not exist" containerID="10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308" Jul 29 21:46:25.835089 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.834965 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308"} err="failed to get container status \"10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308\": rpc error: code = NotFound desc = could not find container \"10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308\": container with ID starting with 10bea02d9c7818495d27177f235f1ddcc94dc65cad03e6257b37816ec4e5a308 not found: ID does not exist" Jul 29 21:46:25.837640 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.837577 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms"] Jul 29 21:46:25.839899 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.839878 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-eb3087-openshift-default-5c69fc9dfb-8xmms"] Jul 29 21:46:25.894476 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.894450 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" Jul 29 21:46:25.896622 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.896601 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" Jul 29 21:46:25.942456 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.942424 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9l5k\" (UniqueName: \"kubernetes.io/projected/5a10363e-a2d1-437f-81c8-842810aad721-kube-api-access-r9l5k\") pod \"5a10363e-a2d1-437f-81c8-842810aad721\" (UID: \"5a10363e-a2d1-437f-81c8-842810aad721\") " Jul 29 21:46:25.942652 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.942489 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8hsj\" (UniqueName: \"kubernetes.io/projected/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b-kube-api-access-v8hsj\") pod \"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b\" (UID: \"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b\") " Jul 29 21:46:25.942652 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.942564 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a10363e-a2d1-437f-81c8-842810aad721-plugins-config-volume\") pod \"5a10363e-a2d1-437f-81c8-842810aad721\" (UID: \"5a10363e-a2d1-437f-81c8-842810aad721\") " Jul 29 21:46:25.942652 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.942591 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b-plugins-config-volume\") pod \"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b\" (UID: \"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b\") " Jul 29 21:46:25.942956 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.942933 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a10363e-a2d1-437f-81c8-842810aad721-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a10363e-a2d1-437f-81c8-842810aad721" (UID: "5a10363e-a2d1-437f-81c8-842810aad721"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:46:25.942956 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.942938 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c345ed65-4a4f-4b61-ac8e-cf0daa593f6b" (UID: "c345ed65-4a4f-4b61-ac8e-cf0daa593f6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:46:25.945378 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.945349 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b-kube-api-access-v8hsj" (OuterVolumeSpecName: "kube-api-access-v8hsj") pod "c345ed65-4a4f-4b61-ac8e-cf0daa593f6b" (UID: "c345ed65-4a4f-4b61-ac8e-cf0daa593f6b"). InnerVolumeSpecName "kube-api-access-v8hsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:46:25.945478 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:25.945355 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a10363e-a2d1-437f-81c8-842810aad721-kube-api-access-r9l5k" (OuterVolumeSpecName: "kube-api-access-r9l5k") pod "5a10363e-a2d1-437f-81c8-842810aad721" (UID: "5a10363e-a2d1-437f-81c8-842810aad721"). InnerVolumeSpecName "kube-api-access-r9l5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:46:26.043269 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.043231 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9l5k\" (UniqueName: \"kubernetes.io/projected/5a10363e-a2d1-437f-81c8-842810aad721-kube-api-access-r9l5k\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:26.043269 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.043266 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8hsj\" (UniqueName: \"kubernetes.io/projected/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b-kube-api-access-v8hsj\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:26.043269 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.043278 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a10363e-a2d1-437f-81c8-842810aad721-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:26.043516 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.043290 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b-plugins-config-volume\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:26.494727 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.494692 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="171461e2-b759-4116-992b-14d0feb8005f" path="/var/lib/kubelet/pods/171461e2-b759-4116-992b-14d0feb8005f/volumes" Jul 29 21:46:26.816451 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.816367 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" event={"ID":"c345ed65-4a4f-4b61-ac8e-cf0daa593f6b","Type":"ContainerDied","Data":"5a0327ed4ef6c96e751956d791bc623072577c130507dceed3e1d0cc51064c5b"} Jul 29 21:46:26.816451 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.816442 2582 scope.go:117] "RemoveContainer" containerID="5f112522c05bf6e85b465f003a5c7aa77648bbd1a5b4dc6097ea64907b13dcf6" Jul 29 21:46:26.816662 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.816373 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" Jul 29 21:46:26.817853 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.817821 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" Jul 29 21:46:26.817941 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.817908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" event={"ID":"5a10363e-a2d1-437f-81c8-842810aad721","Type":"ContainerDied","Data":"daf083ff5cf46a05ed9c09d7b80ac7503502c8ab5927e16639d5379d06959262"} Jul 29 21:46:26.825911 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.825891 2582 scope.go:117] "RemoveContainer" containerID="71ae872f417b24ed27ac1183dbac146a0089d89df21d3280e6714e756a9b1c2e" Jul 29 21:46:26.841746 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.841721 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5"] Jul 29 21:46:26.847470 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.847432 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5"] Jul 29 21:46:26.857710 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.857674 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52"] Jul 29 21:46:26.862057 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.862008 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52"] Jul 29 21:46:26.872923 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.872893 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-46bb38-7c8b4b75c6-rjz52" podUID="c345ed65-4a4f-4b61-ac8e-cf0daa593f6b" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.89:9004: i/o timeout" Jul 29 21:46:26.873082 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:26.872893 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-46bb38-59849c9d86-t7zt5" podUID="5a10363e-a2d1-437f-81c8-842810aad721" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.90:9004: i/o timeout" Jul 29 21:46:28.492750 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:28.492716 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a10363e-a2d1-437f-81c8-842810aad721" path="/var/lib/kubelet/pods/5a10363e-a2d1-437f-81c8-842810aad721/volumes" Jul 29 21:46:28.493173 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:28.493107 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c345ed65-4a4f-4b61-ac8e-cf0daa593f6b" path="/var/lib/kubelet/pods/c345ed65-4a4f-4b61-ac8e-cf0daa593f6b/volumes" Jul 29 21:46:29.057811 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:29.057767 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-79tv9"] Jul 29 21:46:32.807855 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:32.807825 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n"] Jul 29 21:46:32.808302 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:32.808048 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerName="istio-proxy" containerID="cri-o://db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a" gracePeriod=30 Jul 29 21:46:33.514419 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:33.514384 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d5759c778-m8bnm_30e6ac2d-e619-4729-b687-771df9b4ab32/authorino/0.log" Jul 29 21:46:38.058567 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.058539 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:46:38.152044 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.151937 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-token\") pod \"4360fb49-4d3c-497e-a4c3-22e4f05866de\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " Jul 29 21:46:38.152044 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.151985 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-envoy\") pod \"4360fb49-4d3c-497e-a4c3-22e4f05866de\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " Jul 29 21:46:38.152044 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.152007 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cr4m\" (UniqueName: \"kubernetes.io/projected/4360fb49-4d3c-497e-a4c3-22e4f05866de-kube-api-access-7cr4m\") pod \"4360fb49-4d3c-497e-a4c3-22e4f05866de\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " Jul 29 21:46:38.152337 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.152117 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-data\") pod \"4360fb49-4d3c-497e-a4c3-22e4f05866de\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " Jul 29 21:46:38.152337 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.152143 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4360fb49-4d3c-497e-a4c3-22e4f05866de-istiod-ca-cert\") pod \"4360fb49-4d3c-497e-a4c3-22e4f05866de\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " Jul 29 21:46:38.152337 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.152188 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-workload-socket\") pod \"4360fb49-4d3c-497e-a4c3-22e4f05866de\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " Jul 29 21:46:38.152337 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.152211 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-credential-socket\") pod \"4360fb49-4d3c-497e-a4c3-22e4f05866de\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " Jul 29 21:46:38.152337 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.152248 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-podinfo\") pod \"4360fb49-4d3c-497e-a4c3-22e4f05866de\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " Jul 29 21:46:38.152337 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.152287 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-workload-certs\") pod \"4360fb49-4d3c-497e-a4c3-22e4f05866de\" (UID: \"4360fb49-4d3c-497e-a4c3-22e4f05866de\") " Jul 29 21:46:38.152649 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.152449 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4360fb49-4d3c-497e-a4c3-22e4f05866de" (UID: "4360fb49-4d3c-497e-a4c3-22e4f05866de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:46:38.152649 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.152620 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-workload-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:38.152649 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.152532 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4360fb49-4d3c-497e-a4c3-22e4f05866de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4360fb49-4d3c-497e-a4c3-22e4f05866de" (UID: "4360fb49-4d3c-497e-a4c3-22e4f05866de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:46:38.152816 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.152610 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4360fb49-4d3c-497e-a4c3-22e4f05866de" (UID: "4360fb49-4d3c-497e-a4c3-22e4f05866de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:46:38.153006 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.152984 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4360fb49-4d3c-497e-a4c3-22e4f05866de" (UID: "4360fb49-4d3c-497e-a4c3-22e4f05866de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:46:38.153298 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.153277 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-data" (OuterVolumeSpecName: "istio-data") pod "4360fb49-4d3c-497e-a4c3-22e4f05866de" (UID: "4360fb49-4d3c-497e-a4c3-22e4f05866de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:46:38.154473 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.154453 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-token" (OuterVolumeSpecName: "istio-token") pod "4360fb49-4d3c-497e-a4c3-22e4f05866de" (UID: "4360fb49-4d3c-497e-a4c3-22e4f05866de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:46:38.154578 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.154481 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4360fb49-4d3c-497e-a4c3-22e4f05866de-kube-api-access-7cr4m" (OuterVolumeSpecName: "kube-api-access-7cr4m") pod "4360fb49-4d3c-497e-a4c3-22e4f05866de" (UID: "4360fb49-4d3c-497e-a4c3-22e4f05866de"). InnerVolumeSpecName "kube-api-access-7cr4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:46:38.154854 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.154837 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4360fb49-4d3c-497e-a4c3-22e4f05866de" (UID: "4360fb49-4d3c-497e-a4c3-22e4f05866de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:46:38.154972 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.154954 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4360fb49-4d3c-497e-a4c3-22e4f05866de" (UID: "4360fb49-4d3c-497e-a4c3-22e4f05866de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 21:46:38.253329 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.253294 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-data\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:38.253329 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.253324 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4360fb49-4d3c-497e-a4c3-22e4f05866de-istiod-ca-cert\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:38.253329 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.253333 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-credential-socket\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:38.253569 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.253342 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-podinfo\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:38.253569 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.253353 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-workload-certs\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:38.253569 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.253362 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-token\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:38.253569 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.253369 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4360fb49-4d3c-497e-a4c3-22e4f05866de-istio-envoy\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:38.253569 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.253377 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cr4m\" (UniqueName: \"kubernetes.io/projected/4360fb49-4d3c-497e-a4c3-22e4f05866de-kube-api-access-7cr4m\") on node \"ip-10-0-137-160.ec2.internal\" DevicePath \"\"" Jul 29 21:46:38.885236 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.885136 2582 generic.go:356] "Generic (PLEG): container finished" podID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerID="db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a" exitCode=0 Jul 29 21:46:38.885236 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.885194 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" Jul 29 21:46:38.885236 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.885219 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" event={"ID":"4360fb49-4d3c-497e-a4c3-22e4f05866de","Type":"ContainerDied","Data":"db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a"} Jul 29 21:46:38.885521 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.885289 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n" event={"ID":"4360fb49-4d3c-497e-a4c3-22e4f05866de","Type":"ContainerDied","Data":"70aadadf39518fedda9a10568b1beabcbc4b7fa077a59cd5cadab8b294a87e88"} Jul 29 21:46:38.885521 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.885318 2582 scope.go:117] "RemoveContainer" containerID="db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a" Jul 29 21:46:38.894722 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.894704 2582 scope.go:117] "RemoveContainer" containerID="bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692" Jul 29 21:46:38.903109 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.903086 2582 scope.go:117] "RemoveContainer" containerID="db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a" Jul 29 21:46:38.903554 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:46:38.903528 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a\": container with ID starting with db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a not found: ID does not exist" containerID="db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a" Jul 29 21:46:38.903676 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.903566 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a"} err="failed to get container status \"db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a\": rpc error: code = NotFound desc = could not find container \"db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a\": container with ID starting with db3d33b301da67e4c675ba086795a519f53f65d271d0db2b6a41f4c1ad93725a not found: ID does not exist" Jul 29 21:46:38.903676 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.903593 2582 scope.go:117] "RemoveContainer" containerID="bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692" Jul 29 21:46:38.904074 ip-10-0-137-160 kubenswrapper[2582]: E0729 21:46:38.903903 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692\": container with ID starting with bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692 not found: ID does not exist" containerID="bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692" Jul 29 21:46:38.904074 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.903939 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692"} err="failed to get container status \"bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692\": rpc error: code = NotFound desc = could not find container \"bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692\": container with ID starting with bb5e733eb8461298f70216fcfc0847cab18d6c08b6467ce872595831d3f2e692 not found: ID does not exist" Jul 29 21:46:38.906304 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.906286 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n"] Jul 29 21:46:38.910763 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.910742 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-46bb38-openshift-default-859f8c9b84-dxf2n"] Jul 29 21:46:38.963297 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:38.963267 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-twx5p_9dae9081-9a2d-4867-b0ca-8697d8fe9b77/manager/0.log" Jul 29 21:46:39.080920 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:39.080888 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-62m6l_fa1408e5-0576-4fda-aa6b-830280b670f3/manager/0.log" Jul 29 21:46:39.320303 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:39.320277 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nn9t2_e9dfe876-9ede-47bb-87f6-d9d9fd916035/server/0.log" Jul 29 21:46:39.441146 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:39.441113 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8r9ws_28c6a372-aafb-4449-84b9-81a3263ab62d/manager/0.log" Jul 29 21:46:39.555653 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:39.555624 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57645d8769-6tbhp_9e33cf81-e1d6-442f-a673-8a032a0b0b14/copy-manifests/0.log" Jul 29 21:46:39.563247 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:39.563221 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57645d8769-6tbhp_9e33cf81-e1d6-442f-a673-8a032a0b0b14/manager/0.log" Jul 29 21:46:40.273646 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:40.273613 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755995-k45pd_91f06ae2-cdd2-405c-8060-49fd43a31a77/cleanup/0.log" Jul 29 21:46:40.379152 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:40.379114 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756010-qq5qd_753d0899-a6ac-4059-befa-507d10de4609/cleanup/0.log" Jul 29 21:46:40.494194 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:40.494161 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" path="/var/lib/kubelet/pods/4360fb49-4d3c-497e-a4c3-22e4f05866de/volumes" Jul 29 21:46:41.369727 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.369693 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p_bbf76bf4-b259-4481-8b45-4cde2c1a141a/pull/0.log" Jul 29 21:46:41.376604 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.376575 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p_bbf76bf4-b259-4481-8b45-4cde2c1a141a/extract/0.log" Jul 29 21:46:41.382952 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.382927 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p_bbf76bf4-b259-4481-8b45-4cde2c1a141a/util/0.log" Jul 29 21:46:41.489562 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.489529 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj_d3386e07-5cae-4f80-ae1a-179ec7d5ecfa/util/0.log" Jul 29 21:46:41.496622 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.496596 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj_d3386e07-5cae-4f80-ae1a-179ec7d5ecfa/pull/0.log" Jul 29 21:46:41.503620 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.503588 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj_d3386e07-5cae-4f80-ae1a-179ec7d5ecfa/extract/0.log" Jul 29 21:46:41.615637 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.615611 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm_8ad3b3a4-b175-45e4-88f2-cce5dc55bb43/util/0.log" Jul 29 21:46:41.622897 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.622825 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm_8ad3b3a4-b175-45e4-88f2-cce5dc55bb43/pull/0.log" Jul 29 21:46:41.629757 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.629735 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm_8ad3b3a4-b175-45e4-88f2-cce5dc55bb43/extract/0.log" Jul 29 21:46:41.744996 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.744969 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms_41d29d53-b34e-4aa0-9b3e-a313780f825c/util/0.log" Jul 29 21:46:41.752071 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.752036 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms_41d29d53-b34e-4aa0-9b3e-a313780f825c/pull/0.log" Jul 29 21:46:41.758631 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.758610 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms_41d29d53-b34e-4aa0-9b3e-a313780f825c/extract/0.log" Jul 29 21:46:41.885789 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:41.885700 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d5759c778-m8bnm_30e6ac2d-e619-4729-b687-771df9b4ab32/authorino/0.log" Jul 29 21:46:42.009667 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:42.009641 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-dp5xw_3f8958a3-9c15-4a28-8720-0d4ab0cc2988/manager/0.log" Jul 29 21:46:42.135662 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:42.135632 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-dbw66_1a865267-c588-4d02-a253-fa7c0c84b099/manager/0.log" Jul 29 21:46:42.412169 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:42.412142 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-m5zzj_17ec357e-585c-436c-8476-cd3c290a65a9/registry-server/0.log" Jul 29 21:46:42.544569 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:42.544533 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-wj4cz_d6d56a11-13e1-4a78-876b-a56f7267813c/manager/0.log" Jul 29 21:46:42.664401 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:42.664318 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-79tv9_9a24582f-27fe-4c7f-8ee3-092b06e8de12/limitador/0.log" Jul 29 21:46:43.151113 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:43.151082 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk_4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0/istio-proxy/0.log" Jul 29 21:46:43.383460 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:43.383426 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68db9f4dd4-lrl5x_69af1ab8-f326-4137-9b3c-9cb8638ccf25/kube-auth-proxy/0.log" Jul 29 21:46:43.622930 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:43.622896 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-qbhbs_07571bcc-6ebe-4a1b-8749-f072d9e5f6a9/istio-proxy/1.log" Jul 29 21:46:50.375560 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375526 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wkbd4/must-gather-gx4ml"] Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375862 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d631c8d8-d099-4864-a860-c60ec038f5ef" containerName="revoke-keys" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375873 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d631c8d8-d099-4864-a860-c60ec038f5ef" containerName="revoke-keys" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375886 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7bb0446-62f0-435d-bc3e-580f02c6cf8f" containerName="payload-pre-processing" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375891 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7bb0446-62f0-435d-bc3e-580f02c6cf8f" containerName="payload-pre-processing" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375901 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerName="istio-proxy" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375906 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerName="istio-proxy" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375917 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a10363e-a2d1-437f-81c8-842810aad721" containerName="payload-processing" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375922 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a10363e-a2d1-437f-81c8-842810aad721" containerName="payload-processing" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375930 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerName="istio-proxy" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375936 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerName="istio-proxy" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375943 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375948 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375954 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcaed3c-f027-4b54-9ea7-cda91977fd92" containerName="payload-processing" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375959 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcaed3c-f027-4b54-9ea7-cda91977fd92" containerName="payload-processing" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375967 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375973 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375979 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c345ed65-4a4f-4b61-ac8e-cf0daa593f6b" containerName="payload-pre-processing" Jul 29 21:46:50.376016 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.375985 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c345ed65-4a4f-4b61-ac8e-cf0daa593f6b" containerName="payload-pre-processing" Jul 29 21:46:50.376678 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.376074 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" Jul 29 21:46:50.376678 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.376081 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="171461e2-b759-4116-992b-14d0feb8005f" containerName="istio-proxy" Jul 29 21:46:50.376678 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.376087 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerName="istio-proxy" Jul 29 21:46:50.376678 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.376093 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4360fb49-4d3c-497e-a4c3-22e4f05866de" containerName="istio-proxy" Jul 29 21:46:50.376678 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.376098 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d631c8d8-d099-4864-a860-c60ec038f5ef" containerName="revoke-keys" Jul 29 21:46:50.376678 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.376103 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7bb0446-62f0-435d-bc3e-580f02c6cf8f" containerName="payload-pre-processing" Jul 29 21:46:50.376678 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.376108 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c345ed65-4a4f-4b61-ac8e-cf0daa593f6b" containerName="payload-pre-processing" Jul 29 21:46:50.376678 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.376113 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a10363e-a2d1-437f-81c8-842810aad721" containerName="payload-processing" Jul 29 21:46:50.376678 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.376118 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcaed3c-f027-4b54-9ea7-cda91977fd92" containerName="payload-processing" Jul 29 21:46:50.379054 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.379010 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wkbd4/must-gather-gx4ml" Jul 29 21:46:50.383092 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.383071 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wkbd4\"/\"default-dockercfg-m2tq2\"" Jul 29 21:46:50.383262 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.383243 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wkbd4\"/\"kube-root-ca.crt\"" Jul 29 21:46:50.383388 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.383369 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wkbd4\"/\"openshift-service-ca.crt\"" Jul 29 21:46:50.395062 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.395013 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wkbd4/must-gather-gx4ml"] Jul 29 21:46:50.462467 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.462427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a-must-gather-output\") pod \"must-gather-gx4ml\" (UID: \"c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a\") " pod="openshift-must-gather-wkbd4/must-gather-gx4ml" Jul 29 21:46:50.462682 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.462493 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5q8\" (UniqueName: \"kubernetes.io/projected/c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a-kube-api-access-xg5q8\") pod \"must-gather-gx4ml\" (UID: \"c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a\") " pod="openshift-must-gather-wkbd4/must-gather-gx4ml" Jul 29 21:46:50.563550 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.563509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a-must-gather-output\") pod \"must-gather-gx4ml\" (UID: \"c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a\") " pod="openshift-must-gather-wkbd4/must-gather-gx4ml" Jul 29 21:46:50.563746 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.563576 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5q8\" (UniqueName: \"kubernetes.io/projected/c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a-kube-api-access-xg5q8\") pod \"must-gather-gx4ml\" (UID: \"c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a\") " pod="openshift-must-gather-wkbd4/must-gather-gx4ml" Jul 29 21:46:50.563925 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.563904 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a-must-gather-output\") pod \"must-gather-gx4ml\" (UID: \"c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a\") " pod="openshift-must-gather-wkbd4/must-gather-gx4ml" Jul 29 21:46:50.574595 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.574568 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5q8\" (UniqueName: \"kubernetes.io/projected/c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a-kube-api-access-xg5q8\") pod \"must-gather-gx4ml\" (UID: \"c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a\") " pod="openshift-must-gather-wkbd4/must-gather-gx4ml" Jul 29 21:46:50.688758 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.688667 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wkbd4/must-gather-gx4ml" Jul 29 21:46:50.828000 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.827962 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wkbd4/must-gather-gx4ml"] Jul 29 21:46:50.830835 ip-10-0-137-160 kubenswrapper[2582]: W0729 21:46:50.830808 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc08f4d4d_8acf_4aed_9f4e_0f094b1c1a7a.slice/crio-408d94184ba42f19b84218ec3a92ca697f5f3658670a17199d72b3d88441ee60 WatchSource:0}: Error finding container 408d94184ba42f19b84218ec3a92ca697f5f3658670a17199d72b3d88441ee60: Status 404 returned error can't find the container with id 408d94184ba42f19b84218ec3a92ca697f5f3658670a17199d72b3d88441ee60 Jul 29 21:46:50.832873 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.832855 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 21:46:50.953635 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:50.953552 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wkbd4/must-gather-gx4ml" event={"ID":"c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a","Type":"ContainerStarted","Data":"408d94184ba42f19b84218ec3a92ca697f5f3658670a17199d72b3d88441ee60"} Jul 29 21:46:51.964469 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:51.964427 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wkbd4/must-gather-gx4ml" event={"ID":"c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a","Type":"ContainerStarted","Data":"7947958ca37b64fd7121b56f41a23a9cb7da46f6cb3b023ab9e7f579b2ee0502"} Jul 29 21:46:51.964968 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:51.964487 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wkbd4/must-gather-gx4ml" event={"ID":"c08f4d4d-8acf-4aed-9f4e-0f094b1c1a7a","Type":"ContainerStarted","Data":"7f5f418baf619841db68c6415f9ab70ccc0672c2a9d99c8e0ba14ee73f6eaf30"} Jul 29 21:46:51.986951 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:51.986834 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wkbd4/must-gather-gx4ml" podStartSLOduration=1.0953373260000001 podStartE2EDuration="1.986810807s" totalImagesPullingTime="891.473481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:46:51.983054573 +0000 UTC m=+3560.106644406" watchObservedRunningTime="2026-07-29 21:46:51.986810807 +0000 UTC m=+3560.110400651" Jul 29 21:46:53.380535 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:53.380480 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fqq45_2c45ab5d-799b-4ffc-b8e1-40fc891d4d87/global-pull-secret-syncer/0.log" Jul 29 21:46:53.566702 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:53.566658 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v4k8h_fb523307-f7ba-4f4f-8857-a04d1d7f0938/konnectivity-agent/0.log" Jul 29 21:46:53.643636 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:53.643557 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-160.ec2.internal_8e6a9ef8e0380c11ab0ad572fa464511/haproxy/0.log" Jul 29 21:46:58.120116 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:58.120085 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p_bbf76bf4-b259-4481-8b45-4cde2c1a141a/extract/0.log" Jul 29 21:46:58.147601 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:58.147564 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p_bbf76bf4-b259-4481-8b45-4cde2c1a141a/util/0.log" Jul 29 21:46:58.174862 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:58.174826 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwl2p_bbf76bf4-b259-4481-8b45-4cde2c1a141a/pull/0.log" Jul 29 21:46:58.214396 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:58.214348 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj_d3386e07-5cae-4f80-ae1a-179ec7d5ecfa/extract/0.log" Jul 29 21:46:58.240806 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:58.240775 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj_d3386e07-5cae-4f80-ae1a-179ec7d5ecfa/util/0.log" Jul 29 21:46:58.269522 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:58.269489 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k45sj_d3386e07-5cae-4f80-ae1a-179ec7d5ecfa/pull/0.log" Jul 29 21:46:58.304618 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:58.304582 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm_8ad3b3a4-b175-45e4-88f2-cce5dc55bb43/extract/0.log" Jul 29 21:46:58.330014 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:58.329978 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm_8ad3b3a4-b175-45e4-88f2-cce5dc55bb43/util/0.log" Jul 29 21:46:58.359663 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:58.359627 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739chrm_8ad3b3a4-b175-45e4-88f2-cce5dc55bb43/pull/0.log" Jul 29 21:46:58.385422 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:58.385312 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms_41d29d53-b34e-4aa0-9b3e-a313780f825c/extract/0.log" Jul 29 21:46:58.408716 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:58.408683 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms_41d29d53-b34e-4aa0-9b3e-a313780f825c/util/0.log" Jul 29 21:46:58.453306 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:58.453260 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1knzms_41d29d53-b34e-4aa0-9b3e-a313780f825c/pull/0.log" Jul 29 21:46:59.077979 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:59.077363 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d5759c778-m8bnm_30e6ac2d-e619-4729-b687-771df9b4ab32/authorino/0.log" Jul 29 21:46:59.121796 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:59.121759 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-dp5xw_3f8958a3-9c15-4a28-8720-0d4ab0cc2988/manager/0.log" Jul 29 21:46:59.155480 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:59.155447 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-dbw66_1a865267-c588-4d02-a253-fa7c0c84b099/manager/0.log" Jul 29 21:46:59.249513 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:59.249473 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-m5zzj_17ec357e-585c-436c-8476-cd3c290a65a9/registry-server/0.log" Jul 29 21:46:59.543597 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:59.543553 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-wj4cz_d6d56a11-13e1-4a78-876b-a56f7267813c/manager/0.log" Jul 29 21:46:59.579957 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:46:59.579914 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-79tv9_9a24582f-27fe-4c7f-8ee3-092b06e8de12/limitador/0.log" Jul 29 21:47:01.866453 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:01.866416 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-54ff6744d8-bbw8d_b44b094a-135d-4683-a0d4-a2044b2ab112/metrics-server/0.log" Jul 29 21:47:01.951790 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:01.951762 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2f4sz_7d8f5523-e7fb-44e6-8ac2-271a16cf4443/node-exporter/0.log" Jul 29 21:47:01.992630 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:01.992588 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2f4sz_7d8f5523-e7fb-44e6-8ac2-271a16cf4443/kube-rbac-proxy/0.log" Jul 29 21:47:02.032158 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:02.032126 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2f4sz_7d8f5523-e7fb-44e6-8ac2-271a16cf4443/init-textfile/0.log" Jul 29 21:47:03.566850 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.566808 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr"] Jul 29 21:47:03.571969 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.571939 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.585895 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.585858 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr"] Jul 29 21:47:03.705232 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.705182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da7e10b9-7824-464d-bc70-af99e9ad2652-podres\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.705423 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.705283 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da7e10b9-7824-464d-bc70-af99e9ad2652-sys\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.705423 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.705334 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbksp\" (UniqueName: \"kubernetes.io/projected/da7e10b9-7824-464d-bc70-af99e9ad2652-kube-api-access-mbksp\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.705423 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.705399 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da7e10b9-7824-464d-bc70-af99e9ad2652-proc\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.705573 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.705453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da7e10b9-7824-464d-bc70-af99e9ad2652-lib-modules\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.806414 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.806373 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da7e10b9-7824-464d-bc70-af99e9ad2652-lib-modules\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.806614 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.806418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da7e10b9-7824-464d-bc70-af99e9ad2652-podres\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.806614 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.806472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da7e10b9-7824-464d-bc70-af99e9ad2652-sys\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.806614 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.806500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbksp\" (UniqueName: \"kubernetes.io/projected/da7e10b9-7824-464d-bc70-af99e9ad2652-kube-api-access-mbksp\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.806614 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.806542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da7e10b9-7824-464d-bc70-af99e9ad2652-proc\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.806614 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.806598 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da7e10b9-7824-464d-bc70-af99e9ad2652-sys\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.806614 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.806602 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da7e10b9-7824-464d-bc70-af99e9ad2652-lib-modules\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.806849 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.806616 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da7e10b9-7824-464d-bc70-af99e9ad2652-proc\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.806849 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.806625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da7e10b9-7824-464d-bc70-af99e9ad2652-podres\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.815481 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.815447 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbksp\" (UniqueName: \"kubernetes.io/projected/da7e10b9-7824-464d-bc70-af99e9ad2652-kube-api-access-mbksp\") pod \"perf-node-gather-daemonset-tk7tr\" (UID: \"da7e10b9-7824-464d-bc70-af99e9ad2652\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:03.885987 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:03.885874 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:04.087256 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:04.084412 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr"] Jul 29 21:47:05.075924 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:05.075882 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" event={"ID":"da7e10b9-7824-464d-bc70-af99e9ad2652","Type":"ContainerStarted","Data":"bf6e61e1761829706f1b352ed6e79b44102a6a4814a490b5bcac8fd31b34549e"} Jul 29 21:47:05.076445 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:05.075983 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" event={"ID":"da7e10b9-7824-464d-bc70-af99e9ad2652","Type":"ContainerStarted","Data":"d391e1b8c85ec3bd8e43aea3662a2ccafb79835bc385af31e857d3bcad41df1d"} Jul 29 21:47:05.076667 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:05.076646 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:05.097039 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:05.096973 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" podStartSLOduration=2.096942029 podStartE2EDuration="2.096942029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:47:05.096464039 +0000 UTC m=+3573.220053884" watchObservedRunningTime="2026-07-29 21:47:05.096942029 +0000 UTC m=+3573.220531872" Jul 29 21:47:05.344611 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:05.344520 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-844c59684f-d6x4q_aa245b0e-7f9e-4177-81e0-a8b6815f5e1b/console/0.log" Jul 29 21:47:07.090999 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:07.090966 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vcr4b_bf5858a6-49e8-4fc2-986a-e1d1e1658572/dns/0.log" Jul 29 21:47:07.116671 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:07.116603 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vcr4b_bf5858a6-49e8-4fc2-986a-e1d1e1658572/kube-rbac-proxy/0.log" Jul 29 21:47:07.247975 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:07.247945 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v5f5q_4987df3e-bae4-4828-a79f-2cfac383e2d5/dns-node-resolver/0.log" Jul 29 21:47:07.787423 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:07.787353 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c8c48fd5c-mhvt5_02b94d5a-97fb-4149-ad60-937e3e1f6555/registry/0.log" Jul 29 21:47:07.807520 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:07.807475 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c8c48fd5c-mhvt5_02b94d5a-97fb-4149-ad60-937e3e1f6555/registry/1.log" Jul 29 21:47:07.837779 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:07.837734 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ltkns_72e4f020-94dc-446c-9ff1-20a7b4a2f446/node-ca/0.log" Jul 29 21:47:08.856535 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:08.856499 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk_4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0/istio-proxy/0.log" Jul 29 21:47:09.037748 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:09.037713 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68db9f4dd4-lrl5x_69af1ab8-f326-4137-9b3c-9cb8638ccf25/kube-auth-proxy/0.log" Jul 29 21:47:09.098949 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:09.098916 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-qbhbs_07571bcc-6ebe-4a1b-8749-f072d9e5f6a9/istio-proxy/0.log" Jul 29 21:47:09.156233 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:09.156141 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-qbhbs_07571bcc-6ebe-4a1b-8749-f072d9e5f6a9/istio-proxy/1.log" Jul 29 21:47:10.115697 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:10.115658 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wktqs_ea144d0b-7f3d-4833-a83b-c426404b2b12/serve-healthcheck-canary/0.log" Jul 29 21:47:10.746314 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:10.746282 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hl6cv_75b76f3b-b444-47eb-bd66-67f5de1e2c08/kube-rbac-proxy/0.log" Jul 29 21:47:10.772666 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:10.772633 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hl6cv_75b76f3b-b444-47eb-bd66-67f5de1e2c08/exporter/0.log" Jul 29 21:47:10.801734 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:10.801704 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hl6cv_75b76f3b-b444-47eb-bd66-67f5de1e2c08/extractor/0.log" Jul 29 21:47:12.103142 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:12.103111 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-tk7tr" Jul 29 21:47:13.130630 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:13.130575 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-twx5p_9dae9081-9a2d-4867-b0ca-8697d8fe9b77/manager/0.log" Jul 29 21:47:13.524341 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:13.524187 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-62m6l_fa1408e5-0576-4fda-aa6b-830280b670f3/manager/0.log" Jul 29 21:47:13.707273 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:13.707242 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nn9t2_e9dfe876-9ede-47bb-87f6-d9d9fd916035/server/0.log" Jul 29 21:47:13.834834 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:13.834802 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8r9ws_28c6a372-aafb-4449-84b9-81a3263ab62d/manager/0.log" Jul 29 21:47:13.868294 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:13.868220 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57645d8769-6tbhp_9e33cf81-e1d6-442f-a673-8a032a0b0b14/manager/0.log" Jul 29 21:47:13.905347 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:13.905321 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57645d8769-6tbhp_9e33cf81-e1d6-442f-a673-8a032a0b0b14/copy-manifests/0.log" Jul 29 21:47:14.457692 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:14.457655 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk_4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0/istio-proxy/0.log" Jul 29 21:47:14.624417 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:14.624345 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68db9f4dd4-lrl5x_69af1ab8-f326-4137-9b3c-9cb8638ccf25/kube-auth-proxy/0.log" Jul 29 21:47:14.672578 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:14.672544 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-qbhbs_07571bcc-6ebe-4a1b-8749-f072d9e5f6a9/istio-proxy/0.log" Jul 29 21:47:14.729523 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:14.729484 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-qbhbs_07571bcc-6ebe-4a1b-8749-f072d9e5f6a9/istio-proxy/1.log" Jul 29 21:47:16.307591 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:16.307555 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w5lvk_4f86cc40-8ad9-4593-b2aa-4b72f7d6a8f0/istio-proxy/0.log" Jul 29 21:47:16.474342 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:16.474260 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68db9f4dd4-lrl5x_69af1ab8-f326-4137-9b3c-9cb8638ccf25/kube-auth-proxy/0.log" Jul 29 21:47:16.519213 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:16.519175 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-qbhbs_07571bcc-6ebe-4a1b-8749-f072d9e5f6a9/istio-proxy/0.log" Jul 29 21:47:16.575127 ip-10-0-137-160 kubenswrapper[2582]: I0729 21:47:16.575096 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-qbhbs_07571bcc-6ebe-4a1b-8749-f072d9e5f6a9/istio-proxy/1.log"