Jul 22 16:38:31.612682 ip-10-0-136-75 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 16:38:31.612693 ip-10-0-136-75 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 16:38:31.612703 ip-10-0-136-75 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 16:38:31.613082 ip-10-0-136-75 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 16:38:41.757686 ip-10-0-136-75 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 16:38:41.757704 ip-10-0-136-75 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot bc2dfd366de040e9ad5e39ed04511f80 -- Jul 22 16:40:17.521036 ip-10-0-136-75 systemd[1]: Starting Kubernetes Kubelet... Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.938652 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942497 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942511 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942514 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942521 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942524 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942528 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942530 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942533 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:40:17.957350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942540 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942542 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942545 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942548 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942550 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942553 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942556 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942561 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942564 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942567 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942571 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942580 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942583 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942586 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942592 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942595 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942598 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942601 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942604 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942606 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:40:17.960634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942613 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942617 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942621 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942624 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942626 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942634 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942637 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942640 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942643 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942646 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942653 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942655 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942658 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942661 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942663 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942666 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942673 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942675 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942679 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:40:17.963102 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942683 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942686 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942692 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942695 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942697 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942700 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942703 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942706 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942712 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942716 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942718 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942721 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942723 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942726 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942748 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942756 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942759 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942762 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942765 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942769 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:40:17.966430 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942775 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942779 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942781 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942784 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942786 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942789 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942796 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942799 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942801 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942804 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942807 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942809 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942816 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942819 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942821 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942824 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942826 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942829 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942831 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:40:17.976650 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942838 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942842 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942844 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942847 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942849 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942856 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942859 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942861 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942864 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942866 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942869 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942872 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.942879 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943793 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943806 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943809 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943812 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943815 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943818 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943820 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:40:17.979338 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943827 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943830 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943832 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943835 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943838 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943840 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943843 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943845 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943852 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943854 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943857 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943860 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943862 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943866 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943868 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943875 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943877 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943880 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943882 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943885 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:40:17.982297 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943888 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943895 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943898 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943902 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943904 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943907 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943913 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943916 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943919 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943922 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943925 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943928 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943935 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943938 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943940 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943943 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943946 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943953 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943956 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943958 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:40:17.986328 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943961 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943964 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943967 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943969 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943976 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943979 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943981 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943984 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943987 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943990 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943992 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.943999 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944001 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944004 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944006 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944009 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944012 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944015 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944021 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944023 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:40:17.992435 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944026 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944028 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944031 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944033 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944036 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944039 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944045 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944048 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944056 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944060 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944063 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944066 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944073 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944076 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944078 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944081 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944084 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944087 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944094 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944097 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:40:17.997601 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944099 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944102 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944105 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944107 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944110 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944119 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944122 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944126 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944134 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944138 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944141 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944144 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944613 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944622 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944625 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944628 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944631 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944633 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944642 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:40:18.003801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944645 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944648 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944651 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944655 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944663 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944666 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944671 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944674 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944677 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944683 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944687 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944690 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944694 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944696 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944699 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944706 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944708 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944711 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944714 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:40:18.007659 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944716 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944719 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944722 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944730 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944748 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944752 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944755 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944758 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944760 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944763 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944766 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944769 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944774 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944776 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944783 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944786 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944789 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944792 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944794 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:40:18.013763 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944801 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944803 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944806 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944809 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944811 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944814 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944818 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944824 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944827 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944830 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944833 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944835 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944838 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944845 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944848 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944851 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944853 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944856 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944862 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:40:18.017989 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944865 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944867 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944870 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944872 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944875 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944878 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944886 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944889 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944892 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944894 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944896 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944899 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944905 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944908 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944911 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944913 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944916 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944918 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944924 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944928 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:40:18.028905 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944930 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:40:18.019761 ip-10-0-136-75 systemd[1]: Started Kubernetes Kubelet. Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944933 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944936 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944938 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944945 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944947 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944950 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944952 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944955 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944957 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944963 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944966 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944968 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944971 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.944974 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945066 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945073 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945084 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945090 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945095 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945098 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 16:40:18.034480 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945105 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945110 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945113 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945116 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945120 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945123 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945126 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945129 2577 flags.go:64] FLAG: --cgroup-root="" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945136 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945139 2577 flags.go:64] FLAG: --client-ca-file="" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945143 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945146 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945154 2577 flags.go:64] FLAG: --cluster-domain="" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945157 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945160 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945163 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945167 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945174 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945177 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945181 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945184 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945187 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945194 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945197 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 16:40:18.041448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945201 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945204 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945212 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945215 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945218 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945223 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945226 2577 flags.go:64] FLAG: --enable-server="true" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945234 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945239 2577 flags.go:64] FLAG: --event-burst="100" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945242 2577 flags.go:64] FLAG: --event-qps="50" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945245 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945248 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945255 2577 flags.go:64] FLAG: --eviction-hard="" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945264 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945267 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945270 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945277 2577 flags.go:64] FLAG: --eviction-soft="" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945280 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945285 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945288 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945291 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945298 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945301 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945304 2577 flags.go:64] FLAG: --feature-gates="" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945308 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 16:40:18.046519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945311 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945318 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945321 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945324 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945327 2577 flags.go:64] FLAG: --help="false" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945330 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945352 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945360 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945363 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945367 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945370 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945373 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945381 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945384 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945387 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945391 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945394 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945397 2577 flags.go:64] FLAG: --kube-reserved="" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945404 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945407 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945410 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945413 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945416 2577 flags.go:64] FLAG: --lock-file="" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945419 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 16:40:18.213593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945426 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945430 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945436 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945439 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945442 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945448 2577 flags.go:64] FLAG: --logging-format="text" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945452 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945455 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945458 2577 flags.go:64] FLAG: --manifest-url="" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945461 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945466 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945473 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945477 2577 flags.go:64] FLAG: --max-pods="110" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945480 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945483 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945486 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945497 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945501 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945508 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945511 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945520 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945523 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945530 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 16:40:18.233210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945533 2577 flags.go:64] FLAG: --pod-cidr="" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945537 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945542 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945549 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945553 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945556 2577 flags.go:64] FLAG: --port="10250" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945558 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945561 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01140e99d7d09e5de" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945569 2577 flags.go:64] FLAG: --qos-reserved="" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945572 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945577 2577 flags.go:64] FLAG: --register-node="true" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945580 2577 flags.go:64] FLAG: --register-with-taints="" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945584 2577 flags.go:64] FLAG: --registry-burst="10" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945591 2577 flags.go:64] FLAG: --registry-qps="5" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945594 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945597 2577 flags.go:64] FLAG: --reserved-memory="" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945601 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945604 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945607 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945613 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945616 2577 flags.go:64] FLAG: --runonce="false" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945619 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945622 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945625 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945628 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 16:40:18.235806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945635 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945638 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945642 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945645 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945648 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945656 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945659 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945662 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945665 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945669 2577 flags.go:64] FLAG: --system-cgroups="" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945672 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945681 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945684 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945687 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945692 2577 flags.go:64] FLAG: --tls-min-version="" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945699 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945701 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945705 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945708 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945712 2577 flags.go:64] FLAG: --v="2" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945715 2577 flags.go:64] FLAG: --version="false" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945723 2577 flags.go:64] FLAG: --vmodule="" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945727 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.945730 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945880 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:40:18.238539 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945885 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945889 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945892 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945899 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945901 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945904 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945907 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945910 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945912 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945919 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945921 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945924 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945929 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945931 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945934 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945940 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945943 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945946 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945949 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:40:18.247350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945951 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945954 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945956 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945959 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945961 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945965 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945968 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945970 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945973 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945976 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945978 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945981 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945983 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945986 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945988 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945991 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945993 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945996 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.945998 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946001 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:40:18.249851 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946003 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946006 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946008 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946011 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946014 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946021 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946024 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946026 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946029 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946032 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946039 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946042 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946045 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946051 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946055 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946060 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946063 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946067 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946073 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946078 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:40:18.251612 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946081 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946085 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946087 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946090 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946093 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946098 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946101 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946103 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946109 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946111 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946114 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946120 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946124 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946127 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946130 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946136 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946140 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946146 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946148 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:40:18.253155 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946151 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946156 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946158 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946161 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946164 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946171 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946174 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946178 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946183 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946186 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946191 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946194 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946196 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946199 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946205 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946208 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946211 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946215 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946218 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:40:18.254638 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.946220 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.946951 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.963357 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.963373 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.963393 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.963402 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963457 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963462 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963465 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963468 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963471 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963474 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963478 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963481 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:40:18.255481 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963484 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963486 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963489 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963492 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963494 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963497 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963500 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963503 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963506 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963508 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963511 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963514 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963516 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963519 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963521 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963524 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963526 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963529 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963532 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:40:18.256522 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963535 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963537 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963541 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963543 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963547 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963551 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963554 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963557 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963561 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963564 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963566 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963570 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963574 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963577 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963579 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963583 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963585 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963588 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963590 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:40:18.257350 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963593 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963596 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963599 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963601 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963604 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963607 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963609 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963612 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963614 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963617 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963619 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963622 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963625 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963628 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963631 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963633 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963636 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963639 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963642 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963645 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:40:18.258610 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963647 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963650 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963652 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963655 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963658 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963660 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963663 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963665 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963668 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963670 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963673 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963678 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963681 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963684 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963688 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963691 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963693 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963696 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963699 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963702 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:40:18.259448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963704 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963707 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963710 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963712 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963716 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963719 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963722 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963725 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963728 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963731 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963756 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963760 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963763 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.963768 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963887 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:40:18.260311 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963892 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963895 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963898 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963901 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963904 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963906 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963909 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963911 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963914 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963917 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963919 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963922 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963924 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963927 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963930 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963932 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963935 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963938 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963942 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:40:18.260997 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963945 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963948 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963951 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963954 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963957 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963960 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963963 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963966 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963970 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963973 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963976 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963979 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963982 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963985 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963988 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963991 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.963998 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964001 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964004 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:40:18.261634 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964007 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964009 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964012 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964015 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964018 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964020 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964023 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964025 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964028 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964030 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964033 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964037 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964042 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964044 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964047 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964050 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964052 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964055 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964058 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964060 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:40:18.262461 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964063 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964066 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964069 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964071 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964074 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964076 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964079 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964081 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964084 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964088 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964107 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964110 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964113 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964116 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964119 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964122 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964125 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964128 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964131 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964133 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:40:18.263042 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964136 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964138 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964141 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964143 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964147 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964150 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964152 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964154 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964157 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964159 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964162 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964164 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964167 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964169 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964172 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964175 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964178 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964181 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964183 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:40:18.263623 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:17.964186 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:40:18.264201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.964190 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:40:18.264201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.964296 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 16:40:18.264201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.967361 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 16:40:18.264201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.968727 2577 server.go:1019] "Starting client certificate rotation" Jul 22 16:40:18.264201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.968835 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 16:40:18.264201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.969509 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 16:40:18.264201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.971851 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 16:40:18.264201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.975311 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 16:40:18.264201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.987254 2577 log.go:25] "Validated CRI v1 runtime API" Jul 22 16:40:18.264201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.989262 2577 log.go:25] "Validated CRI v1 image API" Jul 22 16:40:18.264201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.990054 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 16:40:18.264201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.993192 2577 fs.go:135] Filesystem UUIDs: map[224c94b2-83b3-434a-bd15-89069b80cffe:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ea125e95-44c4-4629-8647-c3e754068c6c:/dev/nvme0n1p3] Jul 22 16:40:18.264686 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.993213 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 16:40:18.265060 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.999266 2577 manager.go:216] Machine: {Timestamp:2026-07-22 16:40:17.997975044 +0000 UTC m=+0.354371373 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101641 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f98339f51a550e556ae98ec83038d SystemUUID:ec2f9833-9f51-a550-e556-ae98ec83038d BootID:bc2dfd36-6de0-40e9-ad5e-39ed04511f80 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:46:aa:5b:83:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:46:aa:5b:83:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:75:e5:ee:91:a0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 16:40:18.265060 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.999400 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 16:40:18.265060 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:17.999499 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 16:40:18.265060 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.000473 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 16:40:18.265060 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.002714 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 16:40:18.265060 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.002754 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-75.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 16:40:18.265060 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.003761 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.003772 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.003784 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.004368 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.005666 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.008823 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.008929 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.008956 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.008965 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.008979 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.008990 2577 kubelet.go:398] "Adding apiserver pod source" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.009000 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.009326 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.009341 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.011492 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.012860 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.012903 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.013794 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.013812 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.013821 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.013830 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.013842 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.013850 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.013859 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.013865 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.013871 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.013877 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.013886 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.013898 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.014706 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.014714 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.018897 2577 server.go:1268] "Started kubelet" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.018976 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.019118 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.019181 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.020317 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-75.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.020359 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-75.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.020389 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.020664 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 16:40:18.265356 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.021308 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.026026 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.029349 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.029411 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.029442 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.027529 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-75.ec2.internal.18c4aa229f01c1c7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-75.ec2.internal,UID:ip-10-0-136-75.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-75.ec2.internal,},FirstTimestamp:2026-07-22 16:40:18.018861511 +0000 UTC m=+0.375257821,LastTimestamp:2026-07-22 16:40:18.018861511 +0000 UTC m=+0.375257821,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-75.ec2.internal,}" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.030016 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.030255 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.030271 2577 factory.go:55] Registering systemd factory Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.030275 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.030289 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.030280 2577 factory.go:223] Registration of the systemd container factory successfully Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.030435 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.030445 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.030572 2577 factory.go:153] Registering CRI-O factory Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.030583 2577 factory.go:223] Registration of the crio container factory successfully Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.030606 2577 factory.go:103] Registering Raw factory Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.030626 2577 manager.go:1196] Started watching for new ooms in manager Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.031164 2577 manager.go:349] Starting recovery of all containers Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.034515 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cr44n" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.037099 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-75.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.037211 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.041729 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cr44n" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.045883 2577 manager.go:354] Recovery completed Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.054617 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.056678 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.056720 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.056753 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.057419 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.057424 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.057428 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.057464 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 22 16:40:18.269062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.058260 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.058291 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.058305 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.058687 2577 policy_none.go:49] "None policy: Start" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.058703 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.058716 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.059088 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-75.ec2.internal.18c4aa22a1432360 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-75.ec2.internal,UID:ip-10-0-136-75.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-75.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-75.ec2.internal,},FirstTimestamp:2026-07-22 16:40:18.056700768 +0000 UTC m=+0.413097079,LastTimestamp:2026-07-22 16:40:18.056700768 +0000 UTC m=+0.413097079,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-75.ec2.internal,}" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.059958 2577 policy_none.go:47] "Start" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.100115 2577 manager.go:337] "Starting Device Plugin manager" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.100141 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.100151 2577 server.go:87] "Starting device plugin registration server" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.100379 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.100392 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.100478 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.100552 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.100559 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.101038 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.101080 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.178195 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.179365 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.179404 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.179425 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.179461 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.181756 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.200685 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.202044 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.202075 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.202089 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.202119 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.213392 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.213412 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-75.ec2.internal\": node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:18.270536 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.233904 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.279573 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal"] Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.279660 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.281654 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.281689 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.281705 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.283341 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.283544 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.283575 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.285704 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.285725 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.285747 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.287032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.287052 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.287180 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.287193 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.287202 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.287921 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.287939 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.287952 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.311489 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-75.ec2.internal\" not found" node="ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.316286 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-75.ec2.internal\" not found" node="ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.331615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fccdd49421f01c5ce5ce4ca243020bbf-config\") pod \"kube-apiserver-proxy-ip-10-0-136-75.ec2.internal\" (UID: \"fccdd49421f01c5ce5ce4ca243020bbf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.331641 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/df018e68be8a6f187134dbc0b8b139b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal\" (UID: \"df018e68be8a6f187134dbc0b8b139b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.331658 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df018e68be8a6f187134dbc0b8b139b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal\" (UID: \"df018e68be8a6f187134dbc0b8b139b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.375008 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.334689 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:18.498140 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.432765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/df018e68be8a6f187134dbc0b8b139b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal\" (UID: \"df018e68be8a6f187134dbc0b8b139b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.498140 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.432962 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df018e68be8a6f187134dbc0b8b139b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal\" (UID: \"df018e68be8a6f187134dbc0b8b139b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.498140 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.432980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fccdd49421f01c5ce5ce4ca243020bbf-config\") pod \"kube-apiserver-proxy-ip-10-0-136-75.ec2.internal\" (UID: \"fccdd49421f01c5ce5ce4ca243020bbf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.498140 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.432855 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/df018e68be8a6f187134dbc0b8b139b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal\" (UID: \"df018e68be8a6f187134dbc0b8b139b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.498140 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.433028 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fccdd49421f01c5ce5ce4ca243020bbf-config\") pod \"kube-apiserver-proxy-ip-10-0-136-75.ec2.internal\" (UID: \"fccdd49421f01c5ce5ce4ca243020bbf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.498140 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.433074 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df018e68be8a6f187134dbc0b8b139b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal\" (UID: \"df018e68be8a6f187134dbc0b8b139b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.498140 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.434863 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:18.628389 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.535624 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:18.628389 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.615001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.628389 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.619907 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal" Jul 22 16:40:18.758379 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.636305 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:18.758379 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.736532 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:18.916294 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.837455 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:19.070409 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:18.937967 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:19.070409 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.969547 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 16:40:19.070409 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:18.969697 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 16:40:19.070409 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:19.038799 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:19.070409 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:19.044640 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:40:19.070409 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:19.053912 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 16:35:18 +0000 UTC" deadline="2028-04-02 10:41:23.252831208 +0000 UTC" Jul 22 16:40:19.070409 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:19.053939 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14874h1m4.198895506s" Jul 22 16:40:19.138908 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:19.138874 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:19.198300 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:19.198271 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccdd49421f01c5ce5ce4ca243020bbf.slice/crio-daebb56139e313a294a1016f64c9fdd1f00a7aeb70aab2822a7c95e4d369edc2 WatchSource:0}: Error finding container daebb56139e313a294a1016f64c9fdd1f00a7aeb70aab2822a7c95e4d369edc2: Status 404 returned error can't find the container with id daebb56139e313a294a1016f64c9fdd1f00a7aeb70aab2822a7c95e4d369edc2 Jul 22 16:40:19.198954 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:19.198931 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf018e68be8a6f187134dbc0b8b139b6.slice/crio-5b763847e83c45862281a1a9ef281580e7dcdb3b6eeb014a48b507ec8635817d WatchSource:0}: Error finding container 5b763847e83c45862281a1a9ef281580e7dcdb3b6eeb014a48b507ec8635817d: Status 404 returned error can't find the container with id 5b763847e83c45862281a1a9ef281580e7dcdb3b6eeb014a48b507ec8635817d Jul 22 16:40:19.205250 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:19.205233 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:40:19.239940 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:19.239835 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-75.ec2.internal\" not found" Jul 22 16:40:19.293693 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:19.293666 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:40:19.311473 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:19.311446 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:40:19.330848 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:19.330785 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" Jul 22 16:40:19.344861 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:19.344842 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 16:40:19.345678 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:19.345665 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal" Jul 22 16:40:19.352132 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:19.352116 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 16:40:20.011451 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.011415 2577 apiserver.go:52] "Watching apiserver" Jul 22 16:40:20.018210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.018176 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 16:40:20.020546 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.020514 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mvtsj","kube-system/konnectivity-agent-d5smg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz","openshift-dns/node-resolver-pp7b8","openshift-multus/multus-additional-cni-plugins-w9vdr","openshift-ovn-kubernetes/ovnkube-node-mlf2v","kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal","openshift-cluster-node-tuning-operator/tuned-pf6bf","openshift-image-registry/node-ca-hxj9q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal","openshift-multus/multus-fkmdr","openshift-multus/network-metrics-daemon-zx8w2","openshift-network-diagnostics/network-check-target-bdf57","openshift-network-operator/iptables-alerter-ptldn"] Jul 22 16:40:20.022170 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.022145 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:20.022283 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.022237 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:20.024033 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.024007 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d5smg" Jul 22 16:40:20.025161 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.025140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.026698 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.026678 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pp7b8" Jul 22 16:40:20.027074 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.027033 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 16:40:20.027162 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.027096 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 16:40:20.027457 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.027431 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pkb2j\"" Jul 22 16:40:20.027457 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.027437 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 16:40:20.027641 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.027631 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jz65p\"" Jul 22 16:40:20.027707 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.027639 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 16:40:20.027707 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.027688 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 16:40:20.028959 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.028098 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.029066 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.028980 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 16:40:20.029310 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.029286 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 16:40:20.029687 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.029666 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xh5fr\"" Jul 22 16:40:20.032251 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.030042 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.032251 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.031433 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 16:40:20.032251 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.031713 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 16:40:20.032251 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.031962 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 16:40:20.032251 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.031979 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:20.032251 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.032043 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 16:40:20.032251 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.032102 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 16:40:20.032251 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.032038 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:20.032830 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.032356 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 16:40:20.032830 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.032641 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7xfxb\"" Jul 22 16:40:20.032960 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.032932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.033372 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.033106 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 16:40:20.033372 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.033175 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8zlgc\"" Jul 22 16:40:20.034675 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.033787 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 16:40:20.034675 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.033819 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 16:40:20.034675 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.034331 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 16:40:20.034675 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.034548 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 16:40:20.034956 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.034809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hxj9q" Jul 22 16:40:20.035376 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.035035 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 16:40:20.036111 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.035735 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c545c\"" Jul 22 16:40:20.036111 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.035776 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:40:20.036111 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.036025 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 16:40:20.036111 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.036034 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ptldn" Jul 22 16:40:20.037488 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.037194 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 16:40:20.037610 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.037526 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.038082 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.037932 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 16:40:20.042521 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.040312 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:40:20.042521 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.040717 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 16:40:20.042521 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.040837 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f6hjb\"" Jul 22 16:40:20.042521 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.040949 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:20.042521 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.041029 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 16:40:20.042521 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.041111 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 16:40:20.042521 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.041271 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rk5gw\"" Jul 22 16:40:20.042521 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.041412 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:20.042521 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.041715 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 16:40:20.042521 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.041816 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fx4tc\"" Jul 22 16:40:20.043173 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.042913 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-slash\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.043173 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.042969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-ovnkube-script-lib\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.043173 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.043002 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a03e0387-4890-4dec-9ac2-25b236bbdf69-hosts-file\") pod \"node-resolver-pp7b8\" (UID: \"a03e0387-4890-4dec-9ac2-25b236bbdf69\") " pod="openshift-dns/node-resolver-pp7b8" Jul 22 16:40:20.043173 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.043094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-cnibin\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.043173 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.043128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqg27\" (UniqueName: \"kubernetes.io/projected/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-kube-api-access-gqg27\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.043173 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.043163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-systemd-units\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.043644 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.043299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.043644 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.043357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.043644 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.043402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-ovn-node-metrics-cert\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.043644 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.043490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.043644 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.043545 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-sys-fs\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.043644 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.043613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-system-cni-dir\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.043959 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.043867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-run-netns\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.044479 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.043936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-etc-openvswitch\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.044479 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d12a195-9286-4276-b484-9b5ea87c58fb-dbus\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:20.044645 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-device-dir\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.044645 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044524 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03e0387-4890-4dec-9ac2-25b236bbdf69-tmp-dir\") pod \"node-resolver-pp7b8\" (UID: \"a03e0387-4890-4dec-9ac2-25b236bbdf69\") " pod="openshift-dns/node-resolver-pp7b8" Jul 22 16:40:20.044645 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044568 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdmwn\" (UniqueName: \"kubernetes.io/projected/a03e0387-4890-4dec-9ac2-25b236bbdf69-kube-api-access-mdmwn\") pod \"node-resolver-pp7b8\" (UID: \"a03e0387-4890-4dec-9ac2-25b236bbdf69\") " pod="openshift-dns/node-resolver-pp7b8" Jul 22 16:40:20.044645 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044596 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.044645 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-kubelet\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.044930 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044664 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-run-systemd\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.044930 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044697 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-node-log\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.044930 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/887dfeb5-3d3e-4a25-90a6-0f9ccbacce37-agent-certs\") pod \"konnectivity-agent-d5smg\" (UID: \"887dfeb5-3d3e-4a25-90a6-0f9ccbacce37\") " pod="kube-system/konnectivity-agent-d5smg" Jul 22 16:40:20.044930 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.044930 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.044930 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:20.044930 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.044930 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-var-lib-openvswitch\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.045326 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-run-openvswitch\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.045326 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.044980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-socket-dir\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.045326 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-run-ovn\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.045326 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-log-socket\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.045326 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-cni-bin\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.045583 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045358 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-cni-netd\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.045583 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-ovnkube-config\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.045583 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045444 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j88lk\" (UniqueName: \"kubernetes.io/projected/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-kube-api-access-j88lk\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.045583 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045506 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d12a195-9286-4276-b484-9b5ea87c58fb-kubelet-config\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:20.045583 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjgnn\" (UniqueName: \"kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn\") pod \"network-check-target-bdf57\" (UID: \"5109af8a-ceac-4234-b0df-553ed7215c66\") " pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:20.045844 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045581 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.045844 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5lqt\" (UniqueName: \"kubernetes.io/projected/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-kube-api-access-g5lqt\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.045844 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-os-release\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.045844 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-env-overrides\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.046023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045845 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/887dfeb5-3d3e-4a25-90a6-0f9ccbacce37-konnectivity-ca\") pod \"konnectivity-agent-d5smg\" (UID: \"887dfeb5-3d3e-4a25-90a6-0f9ccbacce37\") " pod="kube-system/konnectivity-agent-d5smg" Jul 22 16:40:20.046023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.045872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-registration-dir\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.047277 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.046896 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 16:40:20.069893 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.069861 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5g8ms" Jul 22 16:40:20.080269 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.080248 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5g8ms" Jul 22 16:40:20.131546 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.131468 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 16:40:20.146686 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.146648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-cnibin\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.146686 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.146687 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.146979 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.146713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.146979 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.146738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-ovn-node-metrics-cert\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.146979 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.146840 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.146979 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.146841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-multus-socket-dir-parent\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.146979 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.146882 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-run\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.146979 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.146920 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.146979 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.146920 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-cnibin\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.147330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.146978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-system-cni-dir\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.147330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-run-netns\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.147330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg22h\" (UniqueName: \"kubernetes.io/projected/0effe5a9-83f4-44ab-b1eb-db28c5c445be-kube-api-access-lg22h\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.147330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03e0387-4890-4dec-9ac2-25b236bbdf69-tmp-dir\") pod \"node-resolver-pp7b8\" (UID: \"a03e0387-4890-4dec-9ac2-25b236bbdf69\") " pod="openshift-dns/node-resolver-pp7b8" Jul 22 16:40:20.147330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdmwn\" (UniqueName: \"kubernetes.io/projected/a03e0387-4890-4dec-9ac2-25b236bbdf69-kube-api-access-mdmwn\") pod \"node-resolver-pp7b8\" (UID: \"a03e0387-4890-4dec-9ac2-25b236bbdf69\") " pod="openshift-dns/node-resolver-pp7b8" Jul 22 16:40:20.147330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.147330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-run-systemd\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.147330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfd7f86a-055f-4b35-b7ba-52acb954481a-cni-binary-copy\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.147330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/887dfeb5-3d3e-4a25-90a6-0f9ccbacce37-agent-certs\") pod \"konnectivity-agent-d5smg\" (UID: \"887dfeb5-3d3e-4a25-90a6-0f9ccbacce37\") " pod="kube-system/konnectivity-agent-d5smg" Jul 22 16:40:20.147853 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147361 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-run-multus-certs\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.147853 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147379 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 16:40:20.147853 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147436 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-run-openvswitch\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.147853 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147829 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03e0387-4890-4dec-9ac2-25b236bbdf69-tmp-dir\") pod \"node-resolver-pp7b8\" (UID: \"a03e0387-4890-4dec-9ac2-25b236bbdf69\") " pod="openshift-dns/node-resolver-pp7b8" Jul 22 16:40:20.148067 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147908 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-run-netns\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.148067 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147967 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-system-cni-dir\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.148171 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148085 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.148249 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148231 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-run-systemd\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.147395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-run-openvswitch\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-ovnkube-config\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148410 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d12a195-9286-4276-b484-9b5ea87c58fb-kubelet-config\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-os-release\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148464 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgnn\" (UniqueName: \"kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn\") pod \"network-check-target-bdf57\" (UID: \"5109af8a-ceac-4234-b0df-553ed7215c66\") " pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5lqt\" (UniqueName: \"kubernetes.io/projected/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-kube-api-access-g5lqt\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-sysctl-conf\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148546 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d12a195-9286-4276-b484-9b5ea87c58fb-kubelet-config\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-registration-dir\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-slash\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148599 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-registration-dir\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-etc-kubernetes\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.149865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148667 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148708 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ks7j\" (UniqueName: \"kubernetes.io/projected/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-kube-api-access-6ks7j\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148734 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-sysconfig\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-systemd\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-host\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148846 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-tuned\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a03e0387-4890-4dec-9ac2-25b236bbdf69-hosts-file\") pod \"node-resolver-pp7b8\" (UID: \"a03e0387-4890-4dec-9ac2-25b236bbdf69\") " pod="openshift-dns/node-resolver-pp7b8" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148913 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqg27\" (UniqueName: \"kubernetes.io/projected/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-kube-api-access-gqg27\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-systemd-units\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgjw\" (UniqueName: \"kubernetes.io/projected/0b7f8612-0c2b-418a-b13b-c2eb91c33b02-kube-api-access-ndgjw\") pod \"iptables-alerter-ptldn\" (UID: \"0b7f8612-0c2b-418a-b13b-c2eb91c33b02\") " pod="openshift-network-operator/iptables-alerter-ptldn" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.148993 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-system-cni-dir\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-multus-conf-dir\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfd7f86a-055f-4b35-b7ba-52acb954481a-multus-daemon-config\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149069 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.150619 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-sys-fs\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-etc-openvswitch\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149151 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-ovnkube-config\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d12a195-9286-4276-b484-9b5ea87c58fb-dbus\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149278 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-slash\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149569 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d12a195-9286-4276-b484-9b5ea87c58fb-dbus\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149625 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a03e0387-4890-4dec-9ac2-25b236bbdf69-hosts-file\") pod \"node-resolver-pp7b8\" (UID: \"a03e0387-4890-4dec-9ac2-25b236bbdf69\") " pod="openshift-dns/node-resolver-pp7b8" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149663 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-systemd-units\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-multus-cni-dir\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-run-netns\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149705 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-sys-fs\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-hostroot\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149756 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149790 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-device-dir\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.151289 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149851 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-etc-openvswitch\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149874 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-kubelet\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-device-dir\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-node-log\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.149994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-kubelet\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-var-lib-cni-multus\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150139 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-modprobe-d\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.150147 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-sysctl-d\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.150238 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret podName:3d12a195-9286-4276-b484-9b5ea87c58fb nodeName:}" failed. No retries permitted until 2026-07-22 16:40:20.650210603 +0000 UTC m=+3.006606904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret") pod "global-pull-secret-syncer-mvtsj" (UID: "3d12a195-9286-4276-b484-9b5ea87c58fb") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150353 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b7f8612-0c2b-418a-b13b-c2eb91c33b02-iptables-alerter-script\") pod \"iptables-alerter-ptldn\" (UID: \"0b7f8612-0c2b-418a-b13b-c2eb91c33b02\") " pod="openshift-network-operator/iptables-alerter-ptldn" Jul 22 16:40:20.152791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdjr9\" (UniqueName: \"kubernetes.io/projected/dfd7f86a-055f-4b35-b7ba-52acb954481a-kube-api-access-zdjr9\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150429 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-node-log\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-kubernetes\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150524 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-sys\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150533 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150550 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0effe5a9-83f4-44ab-b1eb-db28c5c445be-tmp\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-var-lib-openvswitch\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b7f8612-0c2b-418a-b13b-c2eb91c33b02-host-slash\") pod \"iptables-alerter-ptldn\" (UID: \"0b7f8612-0c2b-418a-b13b-c2eb91c33b02\") " pod="openshift-network-operator/iptables-alerter-ptldn" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-run-k8s-cni-cncf-io\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-var-lib-kubelet\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-lib-modules\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-var-lib-kubelet\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-socket-dir\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.153562 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-run-ovn\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-log-socket\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.150994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-cni-bin\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-cni-netd\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151037 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j88lk\" (UniqueName: \"kubernetes.io/projected/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-kube-api-access-j88lk\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/407e45e8-f090-4b24-aa8d-53fabbe789ac-host\") pod \"node-ca-hxj9q\" (UID: \"407e45e8-f090-4b24-aa8d-53fabbe789ac\") " pod="openshift-image-registry/node-ca-hxj9q" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-os-release\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-env-overrides\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/407e45e8-f090-4b24-aa8d-53fabbe789ac-serviceca\") pod \"node-ca-hxj9q\" (UID: \"407e45e8-f090-4b24-aa8d-53fabbe789ac\") " pod="openshift-image-registry/node-ca-hxj9q" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151301 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-cni-netd\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151348 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-socket-dir\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151351 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-run-ovn\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151396 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-host-cni-bin\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.154314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151432 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-log-socket\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151472 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-var-lib-openvswitch\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151559 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zfmw\" (UniqueName: \"kubernetes.io/projected/407e45e8-f090-4b24-aa8d-53fabbe789ac-kube-api-access-6zfmw\") pod \"node-ca-hxj9q\" (UID: \"407e45e8-f090-4b24-aa8d-53fabbe789ac\") " pod="openshift-image-registry/node-ca-hxj9q" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/887dfeb5-3d3e-4a25-90a6-0f9ccbacce37-konnectivity-ca\") pod \"konnectivity-agent-d5smg\" (UID: \"887dfeb5-3d3e-4a25-90a6-0f9ccbacce37\") " pod="kube-system/konnectivity-agent-d5smg" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151609 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-os-release\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-ovnkube-script-lib\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-cnibin\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-var-lib-cni-bin\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.151966 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-env-overrides\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.152205 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/887dfeb5-3d3e-4a25-90a6-0f9ccbacce37-konnectivity-ca\") pod \"konnectivity-agent-d5smg\" (UID: \"887dfeb5-3d3e-4a25-90a6-0f9ccbacce37\") " pod="kube-system/konnectivity-agent-d5smg" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.152395 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-ovnkube-script-lib\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.153069 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-ovn-node-metrics-cert\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.155101 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.154961 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/887dfeb5-3d3e-4a25-90a6-0f9ccbacce37-agent-certs\") pod \"konnectivity-agent-d5smg\" (UID: \"887dfeb5-3d3e-4a25-90a6-0f9ccbacce37\") " pod="kube-system/konnectivity-agent-d5smg" Jul 22 16:40:20.160600 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.160540 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdmwn\" (UniqueName: \"kubernetes.io/projected/a03e0387-4890-4dec-9ac2-25b236bbdf69-kube-api-access-mdmwn\") pod \"node-resolver-pp7b8\" (UID: \"a03e0387-4890-4dec-9ac2-25b236bbdf69\") " pod="openshift-dns/node-resolver-pp7b8" Jul 22 16:40:20.162058 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.161417 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:20.162058 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.161439 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:20.162058 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.161453 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vjgnn for pod openshift-network-diagnostics/network-check-target-bdf57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:20.162058 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.161557 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn podName:5109af8a-ceac-4234-b0df-553ed7215c66 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:20.661538595 +0000 UTC m=+3.017934897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgnn" (UniqueName: "kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn") pod "network-check-target-bdf57" (UID: "5109af8a-ceac-4234-b0df-553ed7215c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:20.164004 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.163943 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j88lk\" (UniqueName: \"kubernetes.io/projected/b5b595c3-bf75-4d65-81e4-af2eafbee3b2-kube-api-access-j88lk\") pod \"ovnkube-node-mlf2v\" (UID: \"b5b595c3-bf75-4d65-81e4-af2eafbee3b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.165117 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.165069 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqg27\" (UniqueName: \"kubernetes.io/projected/9fd9a27a-b50a-4409-9cd4-690c8b6cf906-kube-api-access-gqg27\") pod \"multus-additional-cni-plugins-w9vdr\" (UID: \"9fd9a27a-b50a-4409-9cd4-690c8b6cf906\") " pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.166607 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.166546 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5lqt\" (UniqueName: \"kubernetes.io/projected/dd3c2037-fdd1-47d8-9c13-5113e2dbba7e-kube-api-access-g5lqt\") pod \"aws-ebs-csi-driver-node-swsxz\" (UID: \"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.185169 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.185116 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal" event={"ID":"fccdd49421f01c5ce5ce4ca243020bbf","Type":"ContainerStarted","Data":"daebb56139e313a294a1016f64c9fdd1f00a7aeb70aab2822a7c95e4d369edc2"} Jul 22 16:40:20.188023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.187970 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" event={"ID":"df018e68be8a6f187134dbc0b8b139b6","Type":"ContainerStarted","Data":"5b763847e83c45862281a1a9ef281580e7dcdb3b6eeb014a48b507ec8635817d"} Jul 22 16:40:20.252392 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b7f8612-0c2b-418a-b13b-c2eb91c33b02-iptables-alerter-script\") pod \"iptables-alerter-ptldn\" (UID: \"0b7f8612-0c2b-418a-b13b-c2eb91c33b02\") " pod="openshift-network-operator/iptables-alerter-ptldn" Jul 22 16:40:20.252392 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdjr9\" (UniqueName: \"kubernetes.io/projected/dfd7f86a-055f-4b35-b7ba-52acb954481a-kube-api-access-zdjr9\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-kubernetes\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-sys\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252463 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0effe5a9-83f4-44ab-b1eb-db28c5c445be-tmp\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b7f8612-0c2b-418a-b13b-c2eb91c33b02-host-slash\") pod \"iptables-alerter-ptldn\" (UID: \"0b7f8612-0c2b-418a-b13b-c2eb91c33b02\") " pod="openshift-network-operator/iptables-alerter-ptldn" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252508 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-run-k8s-cni-cncf-io\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-var-lib-kubelet\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-lib-modules\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-var-lib-kubelet\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252587 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-kubernetes\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/407e45e8-f090-4b24-aa8d-53fabbe789ac-host\") pod \"node-ca-hxj9q\" (UID: \"407e45e8-f090-4b24-aa8d-53fabbe789ac\") " pod="openshift-image-registry/node-ca-hxj9q" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/407e45e8-f090-4b24-aa8d-53fabbe789ac-serviceca\") pod \"node-ca-hxj9q\" (UID: \"407e45e8-f090-4b24-aa8d-53fabbe789ac\") " pod="openshift-image-registry/node-ca-hxj9q" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252643 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/407e45e8-f090-4b24-aa8d-53fabbe789ac-host\") pod \"node-ca-hxj9q\" (UID: \"407e45e8-f090-4b24-aa8d-53fabbe789ac\") " pod="openshift-image-registry/node-ca-hxj9q" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zfmw\" (UniqueName: \"kubernetes.io/projected/407e45e8-f090-4b24-aa8d-53fabbe789ac-kube-api-access-6zfmw\") pod \"node-ca-hxj9q\" (UID: \"407e45e8-f090-4b24-aa8d-53fabbe789ac\") " pod="openshift-image-registry/node-ca-hxj9q" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252689 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-sys\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.252692 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-cnibin\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-var-lib-cni-bin\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252740 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-cnibin\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-multus-socket-dir-parent\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-run\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg22h\" (UniqueName: \"kubernetes.io/projected/0effe5a9-83f4-44ab-b1eb-db28c5c445be-kube-api-access-lg22h\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfd7f86a-055f-4b35-b7ba-52acb954481a-cni-binary-copy\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-run-multus-certs\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-os-release\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-sysctl-conf\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.252980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-etc-kubernetes\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ks7j\" (UniqueName: \"kubernetes.io/projected/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-kube-api-access-6ks7j\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253033 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-sysconfig\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253056 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-systemd\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-host\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.253519 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253073 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b7f8612-0c2b-418a-b13b-c2eb91c33b02-iptables-alerter-script\") pod \"iptables-alerter-ptldn\" (UID: \"0b7f8612-0c2b-418a-b13b-c2eb91c33b02\") " pod="openshift-network-operator/iptables-alerter-ptldn" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-tuned\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253148 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/407e45e8-f090-4b24-aa8d-53fabbe789ac-serviceca\") pod \"node-ca-hxj9q\" (UID: \"407e45e8-f090-4b24-aa8d-53fabbe789ac\") " pod="openshift-image-registry/node-ca-hxj9q" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgjw\" (UniqueName: \"kubernetes.io/projected/0b7f8612-0c2b-418a-b13b-c2eb91c33b02-kube-api-access-ndgjw\") pod \"iptables-alerter-ptldn\" (UID: \"0b7f8612-0c2b-418a-b13b-c2eb91c33b02\") " pod="openshift-network-operator/iptables-alerter-ptldn" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253175 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b7f8612-0c2b-418a-b13b-c2eb91c33b02-host-slash\") pod \"iptables-alerter-ptldn\" (UID: \"0b7f8612-0c2b-418a-b13b-c2eb91c33b02\") " pod="openshift-network-operator/iptables-alerter-ptldn" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-system-cni-dir\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253224 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-multus-conf-dir\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253236 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-run-k8s-cni-cncf-io\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfd7f86a-055f-4b35-b7ba-52acb954481a-multus-daemon-config\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253274 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-lib-modules\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253282 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-multus-cni-dir\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253297 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-sysctl-conf\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253328 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-var-lib-kubelet\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-run-netns\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253359 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-hostroot\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.254270 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253377 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-multus-conf-dir\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-multus-socket-dir-parent\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253404 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-hostroot\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253413 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-var-lib-cni-bin\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253424 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-run-netns\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253424 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-etc-kubernetes\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253332 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-multus-cni-dir\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253431 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-run\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253497 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-system-cni-dir\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253595 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-var-lib-kubelet\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-os-release\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.253639 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253644 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-run-multus-certs\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253665 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-sysconfig\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.255023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253697 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-systemd\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.255763 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.253702 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs podName:659ef35b-7c38-49f0-bd4a-a8c99e6f2f38 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:20.753681996 +0000 UTC m=+3.110078293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs") pod "network-metrics-daemon-zx8w2" (UID: "659ef35b-7c38-49f0-bd4a-a8c99e6f2f38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:20.255763 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253713 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-host\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.255763 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-var-lib-cni-multus\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255763 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfd7f86a-055f-4b35-b7ba-52acb954481a-host-var-lib-cni-multus\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255763 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-modprobe-d\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.255763 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-sysctl-d\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.255763 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253884 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-modprobe-d\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.255763 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253924 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-sysctl-d\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.255763 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.253934 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfd7f86a-055f-4b35-b7ba-52acb954481a-multus-daemon-config\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.255763 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.254065 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfd7f86a-055f-4b35-b7ba-52acb954481a-cni-binary-copy\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.256532 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.256502 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0effe5a9-83f4-44ab-b1eb-db28c5c445be-etc-tuned\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.256532 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.256527 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0effe5a9-83f4-44ab-b1eb-db28c5c445be-tmp\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.263299 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.263218 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdjr9\" (UniqueName: \"kubernetes.io/projected/dfd7f86a-055f-4b35-b7ba-52acb954481a-kube-api-access-zdjr9\") pod \"multus-fkmdr\" (UID: \"dfd7f86a-055f-4b35-b7ba-52acb954481a\") " pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.264908 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.264861 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ks7j\" (UniqueName: \"kubernetes.io/projected/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-kube-api-access-6ks7j\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:20.265234 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.265209 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zfmw\" (UniqueName: \"kubernetes.io/projected/407e45e8-f090-4b24-aa8d-53fabbe789ac-kube-api-access-6zfmw\") pod \"node-ca-hxj9q\" (UID: \"407e45e8-f090-4b24-aa8d-53fabbe789ac\") " pod="openshift-image-registry/node-ca-hxj9q" Jul 22 16:40:20.265738 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.265652 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgjw\" (UniqueName: \"kubernetes.io/projected/0b7f8612-0c2b-418a-b13b-c2eb91c33b02-kube-api-access-ndgjw\") pod \"iptables-alerter-ptldn\" (UID: \"0b7f8612-0c2b-418a-b13b-c2eb91c33b02\") " pod="openshift-network-operator/iptables-alerter-ptldn" Jul 22 16:40:20.265738 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.265724 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg22h\" (UniqueName: \"kubernetes.io/projected/0effe5a9-83f4-44ab-b1eb-db28c5c445be-kube-api-access-lg22h\") pod \"tuned-pf6bf\" (UID: \"0effe5a9-83f4-44ab-b1eb-db28c5c445be\") " pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.316461 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.316416 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:40:20.341997 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.341938 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d5smg" Jul 22 16:40:20.352462 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.352430 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" Jul 22 16:40:20.362332 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.362304 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pp7b8" Jul 22 16:40:20.370222 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.370191 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w9vdr" Jul 22 16:40:20.379924 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.379904 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:20.388913 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.388857 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" Jul 22 16:40:20.398663 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.398640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hxj9q" Jul 22 16:40:20.410599 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.410574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ptldn" Jul 22 16:40:20.415727 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.415704 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fkmdr" Jul 22 16:40:20.656973 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.656874 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:20.657140 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.657066 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:20.657191 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.657148 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret podName:3d12a195-9286-4276-b484-9b5ea87c58fb nodeName:}" failed. No retries permitted until 2026-07-22 16:40:21.65712912 +0000 UTC m=+4.013525444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret") pod "global-pull-secret-syncer-mvtsj" (UID: "3d12a195-9286-4276-b484-9b5ea87c58fb") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:20.757715 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.757666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgnn\" (UniqueName: \"kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn\") pod \"network-check-target-bdf57\" (UID: \"5109af8a-ceac-4234-b0df-553ed7215c66\") " pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:20.757960 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:20.757723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:20.757960 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.757862 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:20.757960 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.757888 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:20.757960 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.757904 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vjgnn for pod openshift-network-diagnostics/network-check-target-bdf57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:20.758146 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.757974 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn podName:5109af8a-ceac-4234-b0df-553ed7215c66 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:21.757955548 +0000 UTC m=+4.114351870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgnn" (UniqueName: "kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn") pod "network-check-target-bdf57" (UID: "5109af8a-ceac-4234-b0df-553ed7215c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:20.758146 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.757996 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:20.758146 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:20.758060 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs podName:659ef35b-7c38-49f0-bd4a-a8c99e6f2f38 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:21.758042527 +0000 UTC m=+4.114438828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs") pod "network-metrics-daemon-zx8w2" (UID: "659ef35b-7c38-49f0-bd4a-a8c99e6f2f38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:21.084415 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.084330 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 16:35:20 +0000 UTC" deadline="2028-01-04 23:03:16.466614052 +0000 UTC" Jul 22 16:40:21.084415 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.084361 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12750h22m55.382257006s" Jul 22 16:40:21.155134 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:21.155102 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3c2037_fdd1_47d8_9c13_5113e2dbba7e.slice/crio-55d82484d31134f6856f74b40f32cc97dd7f4fda84eb3d6b499049e7a39ec623 WatchSource:0}: Error finding container 55d82484d31134f6856f74b40f32cc97dd7f4fda84eb3d6b499049e7a39ec623: Status 404 returned error can't find the container with id 55d82484d31134f6856f74b40f32cc97dd7f4fda84eb3d6b499049e7a39ec623 Jul 22 16:40:21.159574 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:21.159545 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd9a27a_b50a_4409_9cd4_690c8b6cf906.slice/crio-59a2246919b26df4a2b57a821976c675117fee0d31554fdf4341f5207c7382dd WatchSource:0}: Error finding container 59a2246919b26df4a2b57a821976c675117fee0d31554fdf4341f5207c7382dd: Status 404 returned error can't find the container with id 59a2246919b26df4a2b57a821976c675117fee0d31554fdf4341f5207c7382dd Jul 22 16:40:21.161566 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:21.161546 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0effe5a9_83f4_44ab_b1eb_db28c5c445be.slice/crio-083b59ccc27fd1eec99cdca5772af65010c4f06878174cb9596628d069774e78 WatchSource:0}: Error finding container 083b59ccc27fd1eec99cdca5772af65010c4f06878174cb9596628d069774e78: Status 404 returned error can't find the container with id 083b59ccc27fd1eec99cdca5772af65010c4f06878174cb9596628d069774e78 Jul 22 16:40:21.163413 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:21.163386 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b595c3_bf75_4d65_81e4_af2eafbee3b2.slice/crio-cc8f5cbf958144100e2819763876de4422b3604ecc776f67d453719b07734935 WatchSource:0}: Error finding container cc8f5cbf958144100e2819763876de4422b3604ecc776f67d453719b07734935: Status 404 returned error can't find the container with id cc8f5cbf958144100e2819763876de4422b3604ecc776f67d453719b07734935 Jul 22 16:40:21.164668 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:21.164640 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7f8612_0c2b_418a_b13b_c2eb91c33b02.slice/crio-871ff3eabd4c2c6d1ba8f4bea7773ec4763e767ac281ece94ff5fa63f1f1f69b WatchSource:0}: Error finding container 871ff3eabd4c2c6d1ba8f4bea7773ec4763e767ac281ece94ff5fa63f1f1f69b: Status 404 returned error can't find the container with id 871ff3eabd4c2c6d1ba8f4bea7773ec4763e767ac281ece94ff5fa63f1f1f69b Jul 22 16:40:21.166207 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:21.165461 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407e45e8_f090_4b24_aa8d_53fabbe789ac.slice/crio-06c0b4724a7a059c11603addcce1e9e3f049a8a87823ae630b994fc150088e4c WatchSource:0}: Error finding container 06c0b4724a7a059c11603addcce1e9e3f049a8a87823ae630b994fc150088e4c: Status 404 returned error can't find the container with id 06c0b4724a7a059c11603addcce1e9e3f049a8a87823ae630b994fc150088e4c Jul 22 16:40:21.166444 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:21.166420 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887dfeb5_3d3e_4a25_90a6_0f9ccbacce37.slice/crio-728743b81f0c7bca1c0caf23bc44d553979eb7969cb086937209ec740a55d75b WatchSource:0}: Error finding container 728743b81f0c7bca1c0caf23bc44d553979eb7969cb086937209ec740a55d75b: Status 404 returned error can't find the container with id 728743b81f0c7bca1c0caf23bc44d553979eb7969cb086937209ec740a55d75b Jul 22 16:40:21.167480 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:21.167361 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd7f86a_055f_4b35_b7ba_52acb954481a.slice/crio-8d383aaf7d32901d6e658c39586b5f12e9bec1c8bb9b08b927148307fe421b29 WatchSource:0}: Error finding container 8d383aaf7d32901d6e658c39586b5f12e9bec1c8bb9b08b927148307fe421b29: Status 404 returned error can't find the container with id 8d383aaf7d32901d6e658c39586b5f12e9bec1c8bb9b08b927148307fe421b29 Jul 22 16:40:21.168448 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:21.168405 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03e0387_4890_4dec_9ac2_25b236bbdf69.slice/crio-53e157dd818e94121b796e1278ba95e8d08651df102eb17690fb0ca3cef43eaf WatchSource:0}: Error finding container 53e157dd818e94121b796e1278ba95e8d08651df102eb17690fb0ca3cef43eaf: Status 404 returned error can't find the container with id 53e157dd818e94121b796e1278ba95e8d08651df102eb17690fb0ca3cef43eaf Jul 22 16:40:21.179861 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.179840 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:21.180004 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:21.179975 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:21.191055 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.191008 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" event={"ID":"b5b595c3-bf75-4d65-81e4-af2eafbee3b2","Type":"ContainerStarted","Data":"cc8f5cbf958144100e2819763876de4422b3604ecc776f67d453719b07734935"} Jul 22 16:40:21.192563 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.192195 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9vdr" event={"ID":"9fd9a27a-b50a-4409-9cd4-690c8b6cf906","Type":"ContainerStarted","Data":"59a2246919b26df4a2b57a821976c675117fee0d31554fdf4341f5207c7382dd"} Jul 22 16:40:21.194113 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.194082 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" event={"ID":"0effe5a9-83f4-44ab-b1eb-db28c5c445be","Type":"ContainerStarted","Data":"083b59ccc27fd1eec99cdca5772af65010c4f06878174cb9596628d069774e78"} Jul 22 16:40:21.195330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.195309 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pp7b8" event={"ID":"a03e0387-4890-4dec-9ac2-25b236bbdf69","Type":"ContainerStarted","Data":"53e157dd818e94121b796e1278ba95e8d08651df102eb17690fb0ca3cef43eaf"} Jul 22 16:40:21.196490 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.196469 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fkmdr" event={"ID":"dfd7f86a-055f-4b35-b7ba-52acb954481a","Type":"ContainerStarted","Data":"8d383aaf7d32901d6e658c39586b5f12e9bec1c8bb9b08b927148307fe421b29"} Jul 22 16:40:21.197865 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.197841 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d5smg" event={"ID":"887dfeb5-3d3e-4a25-90a6-0f9ccbacce37","Type":"ContainerStarted","Data":"728743b81f0c7bca1c0caf23bc44d553979eb7969cb086937209ec740a55d75b"} Jul 22 16:40:21.199239 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.199174 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hxj9q" event={"ID":"407e45e8-f090-4b24-aa8d-53fabbe789ac","Type":"ContainerStarted","Data":"06c0b4724a7a059c11603addcce1e9e3f049a8a87823ae630b994fc150088e4c"} Jul 22 16:40:21.200475 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.200446 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ptldn" event={"ID":"0b7f8612-0c2b-418a-b13b-c2eb91c33b02","Type":"ContainerStarted","Data":"871ff3eabd4c2c6d1ba8f4bea7773ec4763e767ac281ece94ff5fa63f1f1f69b"} Jul 22 16:40:21.201580 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.201559 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" event={"ID":"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e","Type":"ContainerStarted","Data":"55d82484d31134f6856f74b40f32cc97dd7f4fda84eb3d6b499049e7a39ec623"} Jul 22 16:40:21.665439 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.665351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:21.665619 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:21.665493 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:21.665619 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:21.665588 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret podName:3d12a195-9286-4276-b484-9b5ea87c58fb nodeName:}" failed. No retries permitted until 2026-07-22 16:40:23.665560848 +0000 UTC m=+6.021957188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret") pod "global-pull-secret-syncer-mvtsj" (UID: "3d12a195-9286-4276-b484-9b5ea87c58fb") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:21.767712 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.766753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgnn\" (UniqueName: \"kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn\") pod \"network-check-target-bdf57\" (UID: \"5109af8a-ceac-4234-b0df-553ed7215c66\") " pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:21.767712 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:21.766851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:21.767712 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:21.767011 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:21.767712 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:21.767092 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs podName:659ef35b-7c38-49f0-bd4a-a8c99e6f2f38 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:23.767071158 +0000 UTC m=+6.123467463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs") pod "network-metrics-daemon-zx8w2" (UID: "659ef35b-7c38-49f0-bd4a-a8c99e6f2f38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:21.767712 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:21.767582 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:21.767712 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:21.767601 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:21.767712 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:21.767616 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vjgnn for pod openshift-network-diagnostics/network-check-target-bdf57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:21.767712 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:21.767669 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn podName:5109af8a-ceac-4234-b0df-553ed7215c66 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:23.767650569 +0000 UTC m=+6.124046894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgnn" (UniqueName: "kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn") pod "network-check-target-bdf57" (UID: "5109af8a-ceac-4234-b0df-553ed7215c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:22.085656 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:22.085454 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 16:35:20 +0000 UTC" deadline="2028-04-25 05:20:07.225729864 +0000 UTC" Jul 22 16:40:22.085656 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:22.085525 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15420h39m45.140209774s" Jul 22 16:40:22.182477 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:22.181941 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:22.182477 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:22.182104 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:22.183001 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:22.182837 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:22.183001 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:22.182950 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:22.221960 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:22.221926 2577 generic.go:356] "Generic (PLEG): container finished" podID="df018e68be8a6f187134dbc0b8b139b6" containerID="766fbfaf25f9b9356fe1229017db9a49395f7e68647b1619ea16b1c23115dcd5" exitCode=0 Jul 22 16:40:22.222127 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:22.222020 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" event={"ID":"df018e68be8a6f187134dbc0b8b139b6","Type":"ContainerDied","Data":"766fbfaf25f9b9356fe1229017db9a49395f7e68647b1619ea16b1c23115dcd5"} Jul 22 16:40:22.238090 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:22.238057 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal" event={"ID":"fccdd49421f01c5ce5ce4ca243020bbf","Type":"ContainerStarted","Data":"447ace8a440cd4b6881251a5faff1d65739d331e0e124bbf88141c0bbd8b0866"} Jul 22 16:40:23.181075 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:23.180988 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:23.181581 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:23.181132 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:23.272500 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:23.272460 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" event={"ID":"df018e68be8a6f187134dbc0b8b139b6","Type":"ContainerStarted","Data":"e93124fac89adce0a796e182b3e82a1b1213eb806fb3c55b212d3904fb2b28f6"} Jul 22 16:40:23.294192 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:23.294123 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-75.ec2.internal" podStartSLOduration=4.29409812 podStartE2EDuration="4.29409812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:22.25674345 +0000 UTC m=+4.613139764" watchObservedRunningTime="2026-07-22 16:40:23.29409812 +0000 UTC m=+5.650494443" Jul 22 16:40:23.294413 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:23.294379 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-75.ec2.internal" podStartSLOduration=4.294369443 podStartE2EDuration="4.294369443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:23.292555762 +0000 UTC m=+5.648952112" watchObservedRunningTime="2026-07-22 16:40:23.294369443 +0000 UTC m=+5.650765768" Jul 22 16:40:23.688178 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:23.688024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:23.688364 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:23.688201 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:23.688364 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:23.688277 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret podName:3d12a195-9286-4276-b484-9b5ea87c58fb nodeName:}" failed. No retries permitted until 2026-07-22 16:40:27.688255843 +0000 UTC m=+10.044652232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret") pod "global-pull-secret-syncer-mvtsj" (UID: "3d12a195-9286-4276-b484-9b5ea87c58fb") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:23.790787 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:23.789008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgnn\" (UniqueName: \"kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn\") pod \"network-check-target-bdf57\" (UID: \"5109af8a-ceac-4234-b0df-553ed7215c66\") " pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:23.790787 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:23.789078 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:23.790787 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:23.789230 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:23.790787 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:23.789292 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs podName:659ef35b-7c38-49f0-bd4a-a8c99e6f2f38 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:27.789272914 +0000 UTC m=+10.145669232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs") pod "network-metrics-daemon-zx8w2" (UID: "659ef35b-7c38-49f0-bd4a-a8c99e6f2f38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:23.790787 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:23.789734 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:23.790787 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:23.789756 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:23.790787 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:23.789770 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vjgnn for pod openshift-network-diagnostics/network-check-target-bdf57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:23.790787 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:23.789832 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn podName:5109af8a-ceac-4234-b0df-553ed7215c66 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:27.789802237 +0000 UTC m=+10.146198555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgnn" (UniqueName: "kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn") pod "network-check-target-bdf57" (UID: "5109af8a-ceac-4234-b0df-553ed7215c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:24.181008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:24.180253 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:24.181008 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:24.180399 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:24.181008 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:24.180253 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:24.181008 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:24.180899 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:25.180659 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:25.180618 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:25.180831 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:25.180788 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:26.179969 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:26.179927 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:26.180401 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:26.180076 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:26.180401 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:26.180262 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:26.180401 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:26.180376 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:27.180282 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:27.180245 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:27.180876 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:27.180400 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:27.724165 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:27.724123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:27.724378 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:27.724281 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:27.724378 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:27.724354 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret podName:3d12a195-9286-4276-b484-9b5ea87c58fb nodeName:}" failed. No retries permitted until 2026-07-22 16:40:35.724334543 +0000 UTC m=+18.080730862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret") pod "global-pull-secret-syncer-mvtsj" (UID: "3d12a195-9286-4276-b484-9b5ea87c58fb") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:27.825586 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:27.825011 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgnn\" (UniqueName: \"kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn\") pod \"network-check-target-bdf57\" (UID: \"5109af8a-ceac-4234-b0df-553ed7215c66\") " pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:27.825586 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:27.825083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:27.825586 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:27.825253 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:27.825586 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:27.825318 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs podName:659ef35b-7c38-49f0-bd4a-a8c99e6f2f38 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:35.825297801 +0000 UTC m=+18.181694098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs") pod "network-metrics-daemon-zx8w2" (UID: "659ef35b-7c38-49f0-bd4a-a8c99e6f2f38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:27.825586 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:27.825460 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:27.825586 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:27.825478 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:27.825586 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:27.825490 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vjgnn for pod openshift-network-diagnostics/network-check-target-bdf57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:27.825586 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:27.825534 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn podName:5109af8a-ceac-4234-b0df-553ed7215c66 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:35.825521196 +0000 UTC m=+18.181917501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgnn" (UniqueName: "kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn") pod "network-check-target-bdf57" (UID: "5109af8a-ceac-4234-b0df-553ed7215c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:28.182605 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:28.181890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:28.182605 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:28.182005 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:28.182605 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:28.182382 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:28.182605 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:28.182490 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:29.180172 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:29.180094 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:29.180407 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:29.180229 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:30.180517 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:30.180481 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:30.180987 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:30.180620 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:30.181284 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:30.181252 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:30.181383 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:30.181363 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:31.179815 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:31.179783 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:31.180098 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:31.179908 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:32.180271 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:32.180240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:32.180668 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:32.180287 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:32.180668 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:32.180374 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:32.180668 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:32.180523 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:33.180319 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:33.180279 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:33.180756 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:33.180393 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:34.180281 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:34.180242 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:34.180476 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:34.180242 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:34.180476 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:34.180382 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:34.180902 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:34.180484 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:35.180119 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:35.180087 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:35.180316 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:35.180213 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:35.784137 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:35.784099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:35.784553 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:35.784238 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:35.784553 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:35.784298 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret podName:3d12a195-9286-4276-b484-9b5ea87c58fb nodeName:}" failed. No retries permitted until 2026-07-22 16:40:51.784280758 +0000 UTC m=+34.140677057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret") pod "global-pull-secret-syncer-mvtsj" (UID: "3d12a195-9286-4276-b484-9b5ea87c58fb") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:35.884808 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:35.884773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgnn\" (UniqueName: \"kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn\") pod \"network-check-target-bdf57\" (UID: \"5109af8a-ceac-4234-b0df-553ed7215c66\") " pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:35.884955 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:35.884834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:35.885017 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:35.884952 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:35.885017 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:35.884966 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:35.885017 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:35.884976 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:35.885017 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:35.884987 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vjgnn for pod openshift-network-diagnostics/network-check-target-bdf57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:35.885147 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:35.885028 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs podName:659ef35b-7c38-49f0-bd4a-a8c99e6f2f38 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:51.88500832 +0000 UTC m=+34.241404619 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs") pod "network-metrics-daemon-zx8w2" (UID: "659ef35b-7c38-49f0-bd4a-a8c99e6f2f38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:35.885147 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:35.885045 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn podName:5109af8a-ceac-4234-b0df-553ed7215c66 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:51.885035709 +0000 UTC m=+34.241432011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgnn" (UniqueName: "kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn") pod "network-check-target-bdf57" (UID: "5109af8a-ceac-4234-b0df-553ed7215c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:36.180342 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:36.180308 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:36.180503 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:36.180431 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:36.180763 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:36.180307 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:36.180763 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:36.180712 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:37.180479 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:37.180441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:37.180999 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:37.180564 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:38.184670 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:38.184131 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:38.184670 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:38.184267 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:38.184670 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:38.184407 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:38.185290 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:38.185106 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:39.179944 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:39.179907 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:39.180147 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:39.180116 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:40.180672 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:40.180632 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:40.181146 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:40.180762 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:40.181146 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:40.180797 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:40.181146 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:40.180889 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:41.180444 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.180206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:41.180643 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:41.180493 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:41.312274 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.312236 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" event={"ID":"0effe5a9-83f4-44ab-b1eb-db28c5c445be","Type":"ContainerStarted","Data":"a9a5273270f2e3bd102c7edb405ba68af465971ee35d9dd8a1310bfa32fc3ff4"} Jul 22 16:40:41.313871 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.313836 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pp7b8" event={"ID":"a03e0387-4890-4dec-9ac2-25b236bbdf69","Type":"ContainerStarted","Data":"2eb30bfdc3ca5eb5b5b75d3227eea6ff6be87293b69a5e48b8799d845057a285"} Jul 22 16:40:41.315204 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.315159 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fkmdr" event={"ID":"dfd7f86a-055f-4b35-b7ba-52acb954481a","Type":"ContainerStarted","Data":"00af67921cb0117e36c7f8dc63e32098b72d8aabf39ced2865c6e6599df0a15c"} Jul 22 16:40:41.316590 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.316568 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d5smg" event={"ID":"887dfeb5-3d3e-4a25-90a6-0f9ccbacce37","Type":"ContainerStarted","Data":"12cf85a59372652d7348a38ee0dd072eaeae764830386cf8501151ce29172918"} Jul 22 16:40:41.317915 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.317895 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hxj9q" event={"ID":"407e45e8-f090-4b24-aa8d-53fabbe789ac","Type":"ContainerStarted","Data":"165da83c73230d3065edd92ae44ec007a08baabc3401f1feaf3158baaa1752ff"} Jul 22 16:40:41.319267 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.319248 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" event={"ID":"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e","Type":"ContainerStarted","Data":"c133627e4159eb629e1c1ec05f0a763f963a45ed993f97751ae54489275dd3a1"} Jul 22 16:40:41.322347 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.322321 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" event={"ID":"b5b595c3-bf75-4d65-81e4-af2eafbee3b2","Type":"ContainerStarted","Data":"7752cdb6905592e20f659f9ea9c41518e94744c5c42c626ceeae4d7ca8382fb3"} Jul 22 16:40:41.322434 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.322388 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" event={"ID":"b5b595c3-bf75-4d65-81e4-af2eafbee3b2","Type":"ContainerStarted","Data":"ff09f4d5491b59e4fa1bc5cdbf1a8002823a68a37f61ac446b17bac2cda835f9"} Jul 22 16:40:41.322481 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.322432 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" event={"ID":"b5b595c3-bf75-4d65-81e4-af2eafbee3b2","Type":"ContainerStarted","Data":"8821bafaf16ffbbb82f6accddc580fe0a453e75475e418ac9d86a7a9896dd4c0"} Jul 22 16:40:41.322529 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.322483 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" event={"ID":"b5b595c3-bf75-4d65-81e4-af2eafbee3b2","Type":"ContainerStarted","Data":"9b86351683f155619dc342756599404d295034bbbd58010620370277ffa5f74c"} Jul 22 16:40:41.322579 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.322531 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" event={"ID":"b5b595c3-bf75-4d65-81e4-af2eafbee3b2","Type":"ContainerStarted","Data":"00d167c779b88d483607e88741537e3a73d1217f1bd525ea023796add408c436"} Jul 22 16:40:41.322579 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.322576 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" event={"ID":"b5b595c3-bf75-4d65-81e4-af2eafbee3b2","Type":"ContainerStarted","Data":"757209af4d509d2fca94e3f6ba22daa4db20d1346124cc0761bf8a99357ab5cc"} Jul 22 16:40:41.323954 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.323933 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fd9a27a-b50a-4409-9cd4-690c8b6cf906" containerID="e34413b97fed58c7d73686d888b4ff32cfe9d0a7ee2c88ac8c8ec1f2c8c77cb3" exitCode=0 Jul 22 16:40:41.324052 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.323960 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9vdr" event={"ID":"9fd9a27a-b50a-4409-9cd4-690c8b6cf906","Type":"ContainerDied","Data":"e34413b97fed58c7d73686d888b4ff32cfe9d0a7ee2c88ac8c8ec1f2c8c77cb3"} Jul 22 16:40:41.328991 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.328945 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pf6bf" podStartSLOduration=4.243695448 podStartE2EDuration="23.32893276s" totalImagesPullingTime="19.085237312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:41.328417754 +0000 UTC m=+23.684814150" watchObservedRunningTime="2026-07-22 16:40:41.32893276 +0000 UTC m=+23.685329077" Jul 22 16:40:41.341708 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.341669 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pp7b8" podStartSLOduration=4.304857737 podStartE2EDuration="23.341655903s" totalImagesPullingTime="19.036798166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:41.341591599 +0000 UTC m=+23.697987938" watchObservedRunningTime="2026-07-22 16:40:41.341655903 +0000 UTC m=+23.698052220" Jul 22 16:40:41.396145 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.396091 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hxj9q" podStartSLOduration=4.359807367 podStartE2EDuration="23.396070584s" totalImagesPullingTime="19.036263217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:41.377674582 +0000 UTC m=+23.734070901" watchObservedRunningTime="2026-07-22 16:40:41.396070584 +0000 UTC m=+23.752466903" Jul 22 16:40:41.396320 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.396234 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fkmdr" podStartSLOduration=4.297039263 podStartE2EDuration="23.396224457s" totalImagesPullingTime="19.099185194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:41.396055548 +0000 UTC m=+23.752451866" watchObservedRunningTime="2026-07-22 16:40:41.396224457 +0000 UTC m=+23.752620775" Jul 22 16:40:41.412189 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.412140 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d5smg" podStartSLOduration=8.261337631 podStartE2EDuration="23.412121892s" totalImagesPullingTime="15.150784261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:41.411876184 +0000 UTC m=+23.768272513" watchObservedRunningTime="2026-07-22 16:40:41.412121892 +0000 UTC m=+23.768518210" Jul 22 16:40:41.891395 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:41.891354 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 16:40:42.113955 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:42.113789 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T16:40:41.891378027Z","UUID":"7aa54f6e-3caa-4cb2-986a-1365a4ee9f65","Handler":null,"Name":"","Endpoint":""} Jul 22 16:40:42.115508 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:42.115482 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 16:40:42.115634 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:42.115527 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 16:40:42.180520 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:42.180489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:42.180685 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:42.180614 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:42.180685 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:42.180669 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:42.180821 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:42.180796 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:42.328609 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:42.328564 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" event={"ID":"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e","Type":"ContainerStarted","Data":"0cf99daf438ebfb411b4495f6d431d255f9c54ba1ae15d21ff7a8e1e86862c3d"} Jul 22 16:40:42.330374 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:42.330343 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ptldn" event={"ID":"0b7f8612-0c2b-418a-b13b-c2eb91c33b02","Type":"ContainerStarted","Data":"940f7579d9ba205cd2c4e0dd4dfc919661b02795277a2b35a0fc0f5a6323ce40"} Jul 22 16:40:42.345257 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:42.345212 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ptldn" podStartSLOduration=5.281308213 podStartE2EDuration="24.34519793s" totalImagesPullingTime="19.063889717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:42.344863727 +0000 UTC m=+24.701260045" watchObservedRunningTime="2026-07-22 16:40:42.34519793 +0000 UTC m=+24.701594247" Jul 22 16:40:43.180490 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:43.180433 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:43.180771 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:43.180606 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:43.337709 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:43.337617 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" event={"ID":"b5b595c3-bf75-4d65-81e4-af2eafbee3b2","Type":"ContainerStarted","Data":"110a9c8acdb3659747e72abe12575feb8ceb1c9c2da8f103bcee95e6f9f661a0"} Jul 22 16:40:44.180371 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:44.180332 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:44.180547 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:44.180332 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:44.180547 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:44.180479 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:44.180547 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:44.180534 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:44.778297 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:44.778249 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d5smg" Jul 22 16:40:44.779011 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:44.778990 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d5smg" Jul 22 16:40:45.180046 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:45.180013 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:45.180157 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:45.180124 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:46.180116 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.179924 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:46.180818 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.179985 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:46.180818 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:46.180190 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:46.180818 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:46.180253 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:46.345425 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.345393 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" event={"ID":"dd3c2037-fdd1-47d8-9c13-5113e2dbba7e","Type":"ContainerStarted","Data":"9ab9ce7f25600d5fc27896c49b3fca3d446deedecf46e9ae284a088bb3805bd6"} Jul 22 16:40:46.349265 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.349239 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" event={"ID":"b5b595c3-bf75-4d65-81e4-af2eafbee3b2","Type":"ContainerStarted","Data":"1f90d8b8e6259ef5e87bb805f4c56bb3aba927221acb9ecdf5c87dc0a87f7bf7"} Jul 22 16:40:46.349837 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.349584 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:46.349837 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.349668 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:46.349837 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.349719 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:46.351246 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.351226 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fd9a27a-b50a-4409-9cd4-690c8b6cf906" containerID="45de71babf2d49fce339e47c770dad47ba09f79073ade4a9b4c5bf69cf2095cd" exitCode=0 Jul 22 16:40:46.351330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.351266 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9vdr" event={"ID":"9fd9a27a-b50a-4409-9cd4-690c8b6cf906","Type":"ContainerDied","Data":"45de71babf2d49fce339e47c770dad47ba09f79073ade4a9b4c5bf69cf2095cd"} Jul 22 16:40:46.365627 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.365604 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:46.366685 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.366647 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swsxz" podStartSLOduration=4.510834479 podStartE2EDuration="28.366634551s" totalImagesPullingTime="23.855800072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:46.366355778 +0000 UTC m=+28.722752095" watchObservedRunningTime="2026-07-22 16:40:46.366634551 +0000 UTC m=+28.723030868" Jul 22 16:40:46.371168 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.371137 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:40:46.413368 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:46.413325 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" podStartSLOduration=9.280673479 podStartE2EDuration="28.413311029s" totalImagesPullingTime="19.13263755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:46.413079676 +0000 UTC m=+28.769475991" watchObservedRunningTime="2026-07-22 16:40:46.413311029 +0000 UTC m=+28.769707347" Jul 22 16:40:47.179932 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:47.179902 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:47.180156 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:47.180003 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:47.635795 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:47.635759 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mvtsj"] Jul 22 16:40:47.636049 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:47.635914 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:47.636119 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:47.636061 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:47.636442 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:47.636421 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bdf57"] Jul 22 16:40:47.636522 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:47.636506 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:47.636615 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:47.636598 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:47.637612 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:47.637590 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zx8w2"] Jul 22 16:40:47.637702 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:47.637692 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:47.637807 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:47.637788 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:48.357211 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:48.357178 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fd9a27a-b50a-4409-9cd4-690c8b6cf906" containerID="b13e0c325663603503a50a16cd90e8b5b311c71a4850983df302b94e1b2e220a" exitCode=0 Jul 22 16:40:48.357677 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:48.357218 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9vdr" event={"ID":"9fd9a27a-b50a-4409-9cd4-690c8b6cf906","Type":"ContainerDied","Data":"b13e0c325663603503a50a16cd90e8b5b311c71a4850983df302b94e1b2e220a"} Jul 22 16:40:49.180365 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:49.180337 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:49.180365 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:49.180310 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:49.180365 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:49.180310 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:49.180643 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:49.180467 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:49.180643 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:49.180519 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:49.180643 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:49.180569 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:50.147905 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:50.147691 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:50.243012 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:50.242977 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:50.257896 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:50.257865 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d5smg" Jul 22 16:40:50.258030 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:50.257987 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 16:40:50.258491 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:50.258461 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d5smg" Jul 22 16:40:50.331117 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:50.331026 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:50.364093 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:50.364061 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fd9a27a-b50a-4409-9cd4-690c8b6cf906" containerID="1b65b98abcc4adfdcbf54359d4babf465dcdd7f1ab49d8b440da6e20da3e49a5" exitCode=0 Jul 22 16:40:50.364257 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:50.364137 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9vdr" event={"ID":"9fd9a27a-b50a-4409-9cd4-690c8b6cf906","Type":"ContainerDied","Data":"1b65b98abcc4adfdcbf54359d4babf465dcdd7f1ab49d8b440da6e20da3e49a5"} Jul 22 16:40:51.180036 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:51.179996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:51.180036 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:51.180021 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:51.180426 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:51.179996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:51.180426 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:51.180108 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtsj" podUID="3d12a195-9286-4276-b484-9b5ea87c58fb" Jul 22 16:40:51.180426 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:51.180176 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zx8w2" podUID="659ef35b-7c38-49f0-bd4a-a8c99e6f2f38" Jul 22 16:40:51.180426 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:51.180239 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf57" podUID="5109af8a-ceac-4234-b0df-553ed7215c66" Jul 22 16:40:51.804593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:51.804553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:51.804789 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:51.804732 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:51.804860 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:51.804812 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret podName:3d12a195-9286-4276-b484-9b5ea87c58fb nodeName:}" failed. No retries permitted until 2026-07-22 16:41:23.804797153 +0000 UTC m=+66.161193449 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret") pod "global-pull-secret-syncer-mvtsj" (UID: "3d12a195-9286-4276-b484-9b5ea87c58fb") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:51.905813 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:51.905782 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgnn\" (UniqueName: \"kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn\") pod \"network-check-target-bdf57\" (UID: \"5109af8a-ceac-4234-b0df-553ed7215c66\") " pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:51.905998 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:51.905838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:51.906066 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:51.906041 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:51.906118 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:51.906074 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:51.906118 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:51.906088 2577 projected.go:196] Error preparing data for projected volume kube-api-access-vjgnn for pod openshift-network-diagnostics/network-check-target-bdf57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:51.906218 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:51.906139 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:51.906218 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:51.906148 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn podName:5109af8a-ceac-4234-b0df-553ed7215c66 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:23.906128453 +0000 UTC m=+66.262524763 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgnn" (UniqueName: "kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn") pod "network-check-target-bdf57" (UID: "5109af8a-ceac-4234-b0df-553ed7215c66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:51.906218 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:51.906204 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs podName:659ef35b-7c38-49f0-bd4a-a8c99e6f2f38 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:23.90618633 +0000 UTC m=+66.262582629 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs") pod "network-metrics-daemon-zx8w2" (UID: "659ef35b-7c38-49f0-bd4a-a8c99e6f2f38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:52.937231 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.937198 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-75.ec2.internal" event="NodeReady" Jul 22 16:40:52.937804 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.937351 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 16:40:52.984606 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.984513 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xl265"] Jul 22 16:40:52.985648 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.985617 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:52.986615 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.986596 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xl265" Jul 22 16:40:52.988137 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.988114 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m89nb"] Jul 22 16:40:52.988937 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.988914 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 16:40:52.989039 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.989010 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 16:40:52.989258 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.989240 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n8ggb\"" Jul 22 16:40:52.989810 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.989795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m89nb" Jul 22 16:40:52.992085 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.992035 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 16:40:52.992359 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.992343 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mkm8j\"" Jul 22 16:40:52.992454 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.992433 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 16:40:52.992592 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.992577 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 16:40:52.999791 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:52.999769 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xl265"] Jul 22 16:40:53.004321 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.004297 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m89nb"] Jul 22 16:40:53.069435 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.069400 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:53.114134 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.114100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:53.114330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.114143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-config-volume\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:53.114330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.114201 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-tmp-dir\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:53.114330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.114247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvvc\" (UniqueName: \"kubernetes.io/projected/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-kube-api-access-7tvvc\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:53.114330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.114264 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3786152c-b59d-4c45-a6e1-a3b3ad670056-cert\") pod \"ingress-canary-m89nb\" (UID: \"3786152c-b59d-4c45-a6e1-a3b3ad670056\") " pod="openshift-ingress-canary/ingress-canary-m89nb" Jul 22 16:40:53.114330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.114288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2t6v\" (UniqueName: \"kubernetes.io/projected/3786152c-b59d-4c45-a6e1-a3b3ad670056-kube-api-access-d2t6v\") pod \"ingress-canary-m89nb\" (UID: \"3786152c-b59d-4c45-a6e1-a3b3ad670056\") " pod="openshift-ingress-canary/ingress-canary-m89nb" Jul 22 16:40:53.148209 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.148178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:53.180162 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.180134 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:40:53.180338 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.180177 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:40:53.180338 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.180253 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:40:53.182955 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.182714 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 16:40:53.182955 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.182793 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 16:40:53.182955 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.182854 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vs4sl\"" Jul 22 16:40:53.182955 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.182897 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d5r55\"" Jul 22 16:40:53.183234 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.182992 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 16:40:53.183234 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.183031 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 16:40:53.214963 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.214944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvvc\" (UniqueName: \"kubernetes.io/projected/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-kube-api-access-7tvvc\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:53.215082 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.214973 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3786152c-b59d-4c45-a6e1-a3b3ad670056-cert\") pod \"ingress-canary-m89nb\" (UID: \"3786152c-b59d-4c45-a6e1-a3b3ad670056\") " pod="openshift-ingress-canary/ingress-canary-m89nb" Jul 22 16:40:53.215082 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.215001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2t6v\" (UniqueName: \"kubernetes.io/projected/3786152c-b59d-4c45-a6e1-a3b3ad670056-kube-api-access-d2t6v\") pod \"ingress-canary-m89nb\" (UID: \"3786152c-b59d-4c45-a6e1-a3b3ad670056\") " pod="openshift-ingress-canary/ingress-canary-m89nb" Jul 22 16:40:53.215258 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.215083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:53.215258 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.215113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-config-volume\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:53.215258 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.215145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-tmp-dir\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:53.215421 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:53.215290 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:40:53.215421 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:53.215341 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls podName:146e6ed1-00ed-473d-b3ef-9f50e261b7bf nodeName:}" failed. No retries permitted until 2026-07-22 16:40:53.715327203 +0000 UTC m=+36.071723499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls") pod "dns-default-xl265" (UID: "146e6ed1-00ed-473d-b3ef-9f50e261b7bf") : secret "dns-default-metrics-tls" not found Jul 22 16:40:53.215552 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.215532 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-tmp-dir\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:53.215902 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.215874 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-config-volume\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:53.219937 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.219916 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3786152c-b59d-4c45-a6e1-a3b3ad670056-cert\") pod \"ingress-canary-m89nb\" (UID: \"3786152c-b59d-4c45-a6e1-a3b3ad670056\") " pod="openshift-ingress-canary/ingress-canary-m89nb" Jul 22 16:40:53.225298 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.225278 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvvc\" (UniqueName: \"kubernetes.io/projected/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-kube-api-access-7tvvc\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:53.225385 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.225320 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2t6v\" (UniqueName: \"kubernetes.io/projected/3786152c-b59d-4c45-a6e1-a3b3ad670056-kube-api-access-d2t6v\") pod \"ingress-canary-m89nb\" (UID: \"3786152c-b59d-4c45-a6e1-a3b3ad670056\") " pod="openshift-ingress-canary/ingress-canary-m89nb" Jul 22 16:40:53.307676 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.307596 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m89nb" Jul 22 16:40:53.495772 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.495551 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m89nb"] Jul 22 16:40:53.530272 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.530237 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:53.719593 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:53.719563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:53.719800 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:53.719705 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:40:53.719800 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:53.719792 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls podName:146e6ed1-00ed-473d-b3ef-9f50e261b7bf nodeName:}" failed. No retries permitted until 2026-07-22 16:40:54.719771256 +0000 UTC m=+37.076167554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls") pod "dns-default-xl265" (UID: "146e6ed1-00ed-473d-b3ef-9f50e261b7bf") : secret "dns-default-metrics-tls" not found Jul 22 16:40:54.116261 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:54.116165 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:54.179895 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:54.179846 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:54.728318 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:54.728243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:54.728525 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:54.728428 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:40:54.728525 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:54.728513 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls podName:146e6ed1-00ed-473d-b3ef-9f50e261b7bf nodeName:}" failed. No retries permitted until 2026-07-22 16:40:56.728491103 +0000 UTC m=+39.084887400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls") pod "dns-default-xl265" (UID: "146e6ed1-00ed-473d-b3ef-9f50e261b7bf") : secret "dns-default-metrics-tls" not found Jul 22 16:40:54.870360 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:40:54.870329 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3786152c_b59d_4c45_a6e1_a3b3ad670056.slice/crio-23c32916fd705ceaa865715dc5caf7ef7335fe072e481538d331c78cc6b000be WatchSource:0}: Error finding container 23c32916fd705ceaa865715dc5caf7ef7335fe072e481538d331c78cc6b000be: Status 404 returned error can't find the container with id 23c32916fd705ceaa865715dc5caf7ef7335fe072e481538d331c78cc6b000be Jul 22 16:40:55.378413 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:55.378330 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m89nb" event={"ID":"3786152c-b59d-4c45-a6e1-a3b3ad670056","Type":"ContainerStarted","Data":"23c32916fd705ceaa865715dc5caf7ef7335fe072e481538d331c78cc6b000be"} Jul 22 16:40:55.380724 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:55.380694 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fd9a27a-b50a-4409-9cd4-690c8b6cf906" containerID="401b4965b57a2da1108c76ae96f94bb258bbd83b1263d8813c0bea21ce68c8f1" exitCode=0 Jul 22 16:40:55.380840 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:55.380761 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9vdr" event={"ID":"9fd9a27a-b50a-4409-9cd4-690c8b6cf906","Type":"ContainerDied","Data":"401b4965b57a2da1108c76ae96f94bb258bbd83b1263d8813c0bea21ce68c8f1"} Jul 22 16:40:56.387350 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:56.387306 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fd9a27a-b50a-4409-9cd4-690c8b6cf906" containerID="589bb07017d8b6a7e9c171eae317e0fde73fd1c41a7867d60879f8256cedd359" exitCode=0 Jul 22 16:40:56.387869 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:56.387368 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9vdr" event={"ID":"9fd9a27a-b50a-4409-9cd4-690c8b6cf906","Type":"ContainerDied","Data":"589bb07017d8b6a7e9c171eae317e0fde73fd1c41a7867d60879f8256cedd359"} Jul 22 16:40:56.744467 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:56.744427 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:40:56.744654 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:56.744587 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:40:56.744716 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:56.744670 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls podName:146e6ed1-00ed-473d-b3ef-9f50e261b7bf nodeName:}" failed. No retries permitted until 2026-07-22 16:41:00.744647159 +0000 UTC m=+43.101043456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls") pod "dns-default-xl265" (UID: "146e6ed1-00ed-473d-b3ef-9f50e261b7bf") : secret "dns-default-metrics-tls" not found Jul 22 16:40:57.371350 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:57.371128 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:57.392991 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:57.392951 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9vdr" event={"ID":"9fd9a27a-b50a-4409-9cd4-690c8b6cf906","Type":"ContainerStarted","Data":"d114d1190dc84b63be1b926ece3304f6d03446cf62e977fd65241d0ddac75ef1"} Jul 22 16:40:57.417224 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:57.417144 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w9vdr" podStartSLOduration=9.710471022 podStartE2EDuration="39.417125763s" totalImagesPullingTime="29.706654741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:57.416062876 +0000 UTC m=+39.772459207" watchObservedRunningTime="2026-07-22 16:40:57.417125763 +0000 UTC m=+39.773522080" Jul 22 16:40:57.571992 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:57.571949 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:57.672714 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:57.672626 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:57.773475 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:57.773441 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:58.396552 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:58.396509 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m89nb" event={"ID":"3786152c-b59d-4c45-a6e1-a3b3ad670056","Type":"ContainerStarted","Data":"37f711f3ba88e4db7902c1e8594f6ad6d319290b3ec8d9ef6d2dfef88dc4f785"} Jul 22 16:40:58.413452 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:58.413398 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m89nb" podStartSLOduration=3.98062308 podStartE2EDuration="6.413383394s" totalImagesPullingTime="2.432760314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:58.412313264 +0000 UTC m=+40.768709585" watchObservedRunningTime="2026-07-22 16:40:58.413383394 +0000 UTC m=+40.769779737" Jul 22 16:40:59.607554 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.607515 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bh8r9"] Jul 22 16:40:59.651808 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.651777 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bh8r9"] Jul 22 16:40:59.651965 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.651888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bh8r9" Jul 22 16:40:59.654408 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.654386 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vjrfd\"" Jul 22 16:40:59.655325 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.655309 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 22 16:40:59.655438 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.655420 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 22 16:40:59.665402 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.665380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4rvm\" (UniqueName: \"kubernetes.io/projected/bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b-kube-api-access-m4rvm\") pod \"migrator-7cc9c7ffd7-bh8r9\" (UID: \"bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bh8r9" Jul 22 16:40:59.766168 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.766130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4rvm\" (UniqueName: \"kubernetes.io/projected/bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b-kube-api-access-m4rvm\") pod \"migrator-7cc9c7ffd7-bh8r9\" (UID: \"bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bh8r9" Jul 22 16:40:59.772526 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.772502 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bc66d"] Jul 22 16:40:59.784787 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.784752 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4rvm\" (UniqueName: \"kubernetes.io/projected/bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b-kube-api-access-m4rvm\") pod \"migrator-7cc9c7ffd7-bh8r9\" (UID: \"bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bh8r9" Jul 22 16:40:59.789686 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.789654 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bc66d"] Jul 22 16:40:59.789849 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.789780 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.792137 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.792109 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hs6zh\"" Jul 22 16:40:59.792249 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.792143 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 16:40:59.792294 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.792267 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 16:40:59.792294 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.792272 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 16:40:59.792522 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.792508 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 16:40:59.867151 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.867061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb95755e-21d5-435c-9b16-eeb89295e8d0-crio-socket\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.867151 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.867102 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb95755e-21d5-435c-9b16-eeb89295e8d0-data-volume\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.867151 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.867123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.867151 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.867146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fgmf\" (UniqueName: \"kubernetes.io/projected/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-api-access-7fgmf\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.867381 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.867199 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.961323 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.961286 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bh8r9" Jul 22 16:40:59.967661 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.967632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.967869 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.967678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb95755e-21d5-435c-9b16-eeb89295e8d0-crio-socket\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.967869 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.967703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb95755e-21d5-435c-9b16-eeb89295e8d0-data-volume\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.967869 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.967728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.967869 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.967768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fgmf\" (UniqueName: \"kubernetes.io/projected/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-api-access-7fgmf\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.967869 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:59.967795 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:40:59.967869 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.967801 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb95755e-21d5-435c-9b16-eeb89295e8d0-crio-socket\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.967869 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:59.967870 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:40:59.968167 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:59.967873 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm podName:fb95755e-21d5-435c-9b16-eeb89295e8d0 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:00.467854913 +0000 UTC m=+42.824251222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bc66d" (UID: "fb95755e-21d5-435c-9b16-eeb89295e8d0") : configmap "kube-rbac-proxy" not found Jul 22 16:40:59.968167 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:40:59.967947 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls podName:fb95755e-21d5-435c-9b16-eeb89295e8d0 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:00.467929434 +0000 UTC m=+42.824325734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bc66d" (UID: "fb95755e-21d5-435c-9b16-eeb89295e8d0") : secret "insights-runtime-extractor-tls" not found Jul 22 16:40:59.968167 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.967998 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb95755e-21d5-435c-9b16-eeb89295e8d0-data-volume\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:40:59.976037 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:40:59.976015 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fgmf\" (UniqueName: \"kubernetes.io/projected/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-api-access-7fgmf\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:00.102762 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:00.102718 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bh8r9"] Jul 22 16:41:00.106015 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:41:00.105986 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb8050e_bdd7_4ff8_9fe7_d927e4f5ef5b.slice/crio-1178cf7f34a61e5cd702cb3e7cd6cd23e4cf04ade4cf905c9792a8c5a4a330e3 WatchSource:0}: Error finding container 1178cf7f34a61e5cd702cb3e7cd6cd23e4cf04ade4cf905c9792a8c5a4a330e3: Status 404 returned error can't find the container with id 1178cf7f34a61e5cd702cb3e7cd6cd23e4cf04ade4cf905c9792a8c5a4a330e3 Jul 22 16:41:00.404918 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:00.404886 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bh8r9" event={"ID":"bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b","Type":"ContainerStarted","Data":"1178cf7f34a61e5cd702cb3e7cd6cd23e4cf04ade4cf905c9792a8c5a4a330e3"} Jul 22 16:41:00.472016 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:00.471990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:00.472173 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:00.472123 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:41:00.472173 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:00.472155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:00.472291 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:00.472176 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls podName:fb95755e-21d5-435c-9b16-eeb89295e8d0 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:01.472162478 +0000 UTC m=+43.828558774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bc66d" (UID: "fb95755e-21d5-435c-9b16-eeb89295e8d0") : secret "insights-runtime-extractor-tls" not found Jul 22 16:41:00.472291 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:00.472216 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:41:00.472291 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:00.472271 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm podName:fb95755e-21d5-435c-9b16-eeb89295e8d0 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:01.472255165 +0000 UTC m=+43.828651477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bc66d" (UID: "fb95755e-21d5-435c-9b16-eeb89295e8d0") : configmap "kube-rbac-proxy" not found Jul 22 16:41:00.774225 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:00.774145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:41:00.774601 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:00.774267 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:41:00.774601 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:00.774340 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls podName:146e6ed1-00ed-473d-b3ef-9f50e261b7bf nodeName:}" failed. No retries permitted until 2026-07-22 16:41:08.774318829 +0000 UTC m=+51.130715149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls") pod "dns-default-xl265" (UID: "146e6ed1-00ed-473d-b3ef-9f50e261b7bf") : secret "dns-default-metrics-tls" not found Jul 22 16:41:01.479549 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.479515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:01.479712 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.479568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:01.479712 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:01.479634 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:41:01.479712 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:01.479673 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:41:01.479712 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:01.479709 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm podName:fb95755e-21d5-435c-9b16-eeb89295e8d0 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:03.479691346 +0000 UTC m=+45.836087665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bc66d" (UID: "fb95755e-21d5-435c-9b16-eeb89295e8d0") : configmap "kube-rbac-proxy" not found Jul 22 16:41:01.479876 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:01.479724 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls podName:fb95755e-21d5-435c-9b16-eeb89295e8d0 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:03.479718314 +0000 UTC m=+45.836114610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bc66d" (UID: "fb95755e-21d5-435c-9b16-eeb89295e8d0") : secret "insights-runtime-extractor-tls" not found Jul 22 16:41:01.831660 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.831634 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-rtrxl"] Jul 22 16:41:01.850048 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.850026 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-rtrxl"] Jul 22 16:41:01.850149 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.850110 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-rtrxl" Jul 22 16:41:01.852834 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.852815 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 22 16:41:01.852925 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.852816 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 22 16:41:01.852925 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.852820 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 22 16:41:01.853449 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.853433 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-n4c46\"" Jul 22 16:41:01.853504 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.853433 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 22 16:41:01.881945 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.881925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhv9\" (UniqueName: \"kubernetes.io/projected/4bd39992-333c-42ed-bec2-eb025df5c12e-kube-api-access-hfhv9\") pod \"service-ca-58598695f7-rtrxl\" (UID: \"4bd39992-333c-42ed-bec2-eb025df5c12e\") " pod="openshift-service-ca/service-ca-58598695f7-rtrxl" Jul 22 16:41:01.882056 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.881966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bd39992-333c-42ed-bec2-eb025df5c12e-signing-key\") pod \"service-ca-58598695f7-rtrxl\" (UID: \"4bd39992-333c-42ed-bec2-eb025df5c12e\") " pod="openshift-service-ca/service-ca-58598695f7-rtrxl" Jul 22 16:41:01.882056 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.882023 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bd39992-333c-42ed-bec2-eb025df5c12e-signing-cabundle\") pod \"service-ca-58598695f7-rtrxl\" (UID: \"4bd39992-333c-42ed-bec2-eb025df5c12e\") " pod="openshift-service-ca/service-ca-58598695f7-rtrxl" Jul 22 16:41:01.982888 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.982850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bd39992-333c-42ed-bec2-eb025df5c12e-signing-key\") pod \"service-ca-58598695f7-rtrxl\" (UID: \"4bd39992-333c-42ed-bec2-eb025df5c12e\") " pod="openshift-service-ca/service-ca-58598695f7-rtrxl" Jul 22 16:41:01.983045 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.982926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bd39992-333c-42ed-bec2-eb025df5c12e-signing-cabundle\") pod \"service-ca-58598695f7-rtrxl\" (UID: \"4bd39992-333c-42ed-bec2-eb025df5c12e\") " pod="openshift-service-ca/service-ca-58598695f7-rtrxl" Jul 22 16:41:01.983045 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.982954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhv9\" (UniqueName: \"kubernetes.io/projected/4bd39992-333c-42ed-bec2-eb025df5c12e-kube-api-access-hfhv9\") pod \"service-ca-58598695f7-rtrxl\" (UID: \"4bd39992-333c-42ed-bec2-eb025df5c12e\") " pod="openshift-service-ca/service-ca-58598695f7-rtrxl" Jul 22 16:41:01.983612 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.983592 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bd39992-333c-42ed-bec2-eb025df5c12e-signing-cabundle\") pod \"service-ca-58598695f7-rtrxl\" (UID: \"4bd39992-333c-42ed-bec2-eb025df5c12e\") " pod="openshift-service-ca/service-ca-58598695f7-rtrxl" Jul 22 16:41:01.985330 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.985310 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bd39992-333c-42ed-bec2-eb025df5c12e-signing-key\") pod \"service-ca-58598695f7-rtrxl\" (UID: \"4bd39992-333c-42ed-bec2-eb025df5c12e\") " pod="openshift-service-ca/service-ca-58598695f7-rtrxl" Jul 22 16:41:01.991664 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:01.991640 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhv9\" (UniqueName: \"kubernetes.io/projected/4bd39992-333c-42ed-bec2-eb025df5c12e-kube-api-access-hfhv9\") pod \"service-ca-58598695f7-rtrxl\" (UID: \"4bd39992-333c-42ed-bec2-eb025df5c12e\") " pod="openshift-service-ca/service-ca-58598695f7-rtrxl" Jul 22 16:41:02.159302 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:02.159273 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-rtrxl" Jul 22 16:41:02.298582 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:02.298555 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-rtrxl"] Jul 22 16:41:02.312779 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:41:02.312750 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd39992_333c_42ed_bec2_eb025df5c12e.slice/crio-f0ebe93f42051b66311555ed3e2626a6365929fea0f2e8378f5312de02c0c895 WatchSource:0}: Error finding container f0ebe93f42051b66311555ed3e2626a6365929fea0f2e8378f5312de02c0c895: Status 404 returned error can't find the container with id f0ebe93f42051b66311555ed3e2626a6365929fea0f2e8378f5312de02c0c895 Jul 22 16:41:02.346592 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:02.346571 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pp7b8_a03e0387-4890-4dec-9ac2-25b236bbdf69/dns-node-resolver/0.log" Jul 22 16:41:02.411210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:02.411146 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bh8r9" event={"ID":"bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b","Type":"ContainerStarted","Data":"5c98c9cfc10b1c0b02eb486b53c8f4cbead97715700d8a1930595245b25c7570"} Jul 22 16:41:02.411210 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:02.411202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bh8r9" event={"ID":"bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b","Type":"ContainerStarted","Data":"520eb63f11eb9c6c6cfbd90b20c2121bf028f2b3811b8398d52ad0093fae9bb4"} Jul 22 16:41:02.412304 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:02.412281 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-rtrxl" event={"ID":"4bd39992-333c-42ed-bec2-eb025df5c12e","Type":"ContainerStarted","Data":"f0ebe93f42051b66311555ed3e2626a6365929fea0f2e8378f5312de02c0c895"} Jul 22 16:41:02.431316 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:02.431275 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bh8r9" podStartSLOduration=1.9108082 podStartE2EDuration="3.431263054s" totalImagesPullingTime="1.520454854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:02.431058911 +0000 UTC m=+44.787455229" watchObservedRunningTime="2026-07-22 16:41:02.431263054 +0000 UTC m=+44.787659371" Jul 22 16:41:02.947051 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:02.947027 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hxj9q_407e45e8-f090-4b24-aa8d-53fabbe789ac/node-ca/0.log" Jul 22 16:41:03.491603 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:03.491571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:03.491797 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:03.491641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:03.491797 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:03.491710 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:41:03.491797 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:03.491768 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:41:03.491797 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:03.491784 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls podName:fb95755e-21d5-435c-9b16-eeb89295e8d0 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:07.491768086 +0000 UTC m=+49.848164382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bc66d" (UID: "fb95755e-21d5-435c-9b16-eeb89295e8d0") : secret "insights-runtime-extractor-tls" not found Jul 22 16:41:03.491949 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:03.491818 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm podName:fb95755e-21d5-435c-9b16-eeb89295e8d0 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:07.491806051 +0000 UTC m=+49.848202347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bc66d" (UID: "fb95755e-21d5-435c-9b16-eeb89295e8d0") : configmap "kube-rbac-proxy" not found Jul 22 16:41:03.745972 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:03.745897 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m89nb_3786152c-b59d-4c45-a6e1-a3b3ad670056/serve-healthcheck-canary/0.log" Jul 22 16:41:03.950426 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:03.950398 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-bh8r9_bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b/migrator/0.log" Jul 22 16:41:04.148679 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:04.148652 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-bh8r9_bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b/graceful-termination/0.log" Jul 22 16:41:06.424686 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:06.424651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-rtrxl" event={"ID":"4bd39992-333c-42ed-bec2-eb025df5c12e","Type":"ContainerStarted","Data":"e5eec2a3fd94ddedfac9e69248d44547395456ce6440c7c3b01855431310f137"} Jul 22 16:41:06.447836 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:06.447794 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-rtrxl" podStartSLOduration=2.187255268 podStartE2EDuration="5.447780781s" totalImagesPullingTime="3.260525513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:06.446884746 +0000 UTC m=+48.803281109" watchObservedRunningTime="2026-07-22 16:41:06.447780781 +0000 UTC m=+48.804177099" Jul 22 16:41:07.520440 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:07.520381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:07.520924 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:07.520474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:07.520924 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:07.520573 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:41:07.520924 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:07.520631 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls podName:fb95755e-21d5-435c-9b16-eeb89295e8d0 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:15.520614431 +0000 UTC m=+57.877010734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bc66d" (UID: "fb95755e-21d5-435c-9b16-eeb89295e8d0") : secret "insights-runtime-extractor-tls" not found Jul 22 16:41:07.521426 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:07.521402 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb95755e-21d5-435c-9b16-eeb89295e8d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:08.830591 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:08.830550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:41:08.831030 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:08.830665 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:41:08.831030 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:08.830720 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls podName:146e6ed1-00ed-473d-b3ef-9f50e261b7bf nodeName:}" failed. No retries permitted until 2026-07-22 16:41:24.830706792 +0000 UTC m=+67.187103088 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls") pod "dns-default-xl265" (UID: "146e6ed1-00ed-473d-b3ef-9f50e261b7bf") : secret "dns-default-metrics-tls" not found Jul 22 16:41:15.582950 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:15.582907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:15.585449 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:15.585425 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb95755e-21d5-435c-9b16-eeb89295e8d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bc66d\" (UID: \"fb95755e-21d5-435c-9b16-eeb89295e8d0\") " pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:15.698166 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:15.698137 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bc66d" Jul 22 16:41:15.828485 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:15.828448 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bc66d"] Jul 22 16:41:15.832778 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:41:15.832748 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb95755e_21d5_435c_9b16_eeb89295e8d0.slice/crio-6ccb300b8ee49781e93527b290fc881c903c6b366d68dd975cecf457e38ed848 WatchSource:0}: Error finding container 6ccb300b8ee49781e93527b290fc881c903c6b366d68dd975cecf457e38ed848: Status 404 returned error can't find the container with id 6ccb300b8ee49781e93527b290fc881c903c6b366d68dd975cecf457e38ed848 Jul 22 16:41:16.450583 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:16.450542 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bc66d" event={"ID":"fb95755e-21d5-435c-9b16-eeb89295e8d0","Type":"ContainerStarted","Data":"6ccb300b8ee49781e93527b290fc881c903c6b366d68dd975cecf457e38ed848"} Jul 22 16:41:18.368571 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:18.368536 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlf2v" Jul 22 16:41:21.465530 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:21.465494 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bc66d" event={"ID":"fb95755e-21d5-435c-9b16-eeb89295e8d0","Type":"ContainerStarted","Data":"b60c245fc5d157ea4495094964c0550dca4170a285de5d17a793767050b9ba09"} Jul 22 16:41:23.850712 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:23.850672 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:41:23.853506 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:23.853481 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 16:41:23.864145 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:23.864110 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d12a195-9286-4276-b484-9b5ea87c58fb-original-pull-secret\") pod \"global-pull-secret-syncer-mvtsj\" (UID: \"3d12a195-9286-4276-b484-9b5ea87c58fb\") " pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:41:23.951645 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:23.951604 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgnn\" (UniqueName: \"kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn\") pod \"network-check-target-bdf57\" (UID: \"5109af8a-ceac-4234-b0df-553ed7215c66\") " pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:41:23.951857 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:23.951664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:41:23.954278 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:23.954250 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 16:41:23.954401 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:23.954300 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 16:41:23.964112 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:23.964081 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 16:41:23.964323 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:23.964301 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ef35b-7c38-49f0-bd4a-a8c99e6f2f38-metrics-certs\") pod \"network-metrics-daemon-zx8w2\" (UID: \"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38\") " pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:41:23.975616 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:23.975582 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjgnn\" (UniqueName: \"kubernetes.io/projected/5109af8a-ceac-4234-b0df-553ed7215c66-kube-api-access-vjgnn\") pod \"network-check-target-bdf57\" (UID: \"5109af8a-ceac-4234-b0df-553ed7215c66\") " pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:41:24.094784 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:24.094750 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d5r55\"" Jul 22 16:41:24.099190 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:24.099159 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtsj" Jul 22 16:41:24.102867 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:24.102796 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zx8w2" Jul 22 16:41:24.107976 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:24.107896 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vs4sl\"" Jul 22 16:41:24.115300 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:24.115270 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:41:24.521455 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:24.521315 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mvtsj"] Jul 22 16:41:24.526945 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:41:24.526916 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d12a195_9286_4276_b484_9b5ea87c58fb.slice/crio-4a24d7c9a585eef1c122a9370915983ac6d2e5ce082c2af9bee41c5695afcc1c WatchSource:0}: Error finding container 4a24d7c9a585eef1c122a9370915983ac6d2e5ce082c2af9bee41c5695afcc1c: Status 404 returned error can't find the container with id 4a24d7c9a585eef1c122a9370915983ac6d2e5ce082c2af9bee41c5695afcc1c Jul 22 16:41:24.544725 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:24.544649 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bdf57"] Jul 22 16:41:24.548264 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:41:24.548238 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5109af8a_ceac_4234_b0df_553ed7215c66.slice/crio-cdf8d0ff90b14105f8ae1b4c2344b4093bd286335df1f6eca9e4f43ec6ec21bc WatchSource:0}: Error finding container cdf8d0ff90b14105f8ae1b4c2344b4093bd286335df1f6eca9e4f43ec6ec21bc: Status 404 returned error can't find the container with id cdf8d0ff90b14105f8ae1b4c2344b4093bd286335df1f6eca9e4f43ec6ec21bc Jul 22 16:41:24.560957 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:24.560926 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zx8w2"] Jul 22 16:41:24.564596 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:41:24.564559 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659ef35b_7c38_49f0_bd4a_a8c99e6f2f38.slice/crio-ff7f8ea5055d39753e65b1bfefbb82b5aa278ecc4019ddee2eb2a91d72db79d8 WatchSource:0}: Error finding container ff7f8ea5055d39753e65b1bfefbb82b5aa278ecc4019ddee2eb2a91d72db79d8: Status 404 returned error can't find the container with id ff7f8ea5055d39753e65b1bfefbb82b5aa278ecc4019ddee2eb2a91d72db79d8 Jul 22 16:41:24.858991 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:24.858954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:41:24.861180 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:24.861162 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/146e6ed1-00ed-473d-b3ef-9f50e261b7bf-metrics-tls\") pod \"dns-default-xl265\" (UID: \"146e6ed1-00ed-473d-b3ef-9f50e261b7bf\") " pod="openshift-dns/dns-default-xl265" Jul 22 16:41:25.104148 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:25.104113 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n8ggb\"" Jul 22 16:41:25.110570 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:25.110400 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xl265" Jul 22 16:41:25.290671 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:25.290587 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xl265"] Jul 22 16:41:25.484753 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:25.484673 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bdf57" event={"ID":"5109af8a-ceac-4234-b0df-553ed7215c66","Type":"ContainerStarted","Data":"cdf8d0ff90b14105f8ae1b4c2344b4093bd286335df1f6eca9e4f43ec6ec21bc"} Jul 22 16:41:25.487159 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:25.487071 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zx8w2" event={"ID":"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38","Type":"ContainerStarted","Data":"ff7f8ea5055d39753e65b1bfefbb82b5aa278ecc4019ddee2eb2a91d72db79d8"} Jul 22 16:41:25.489153 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:25.489041 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mvtsj" event={"ID":"3d12a195-9286-4276-b484-9b5ea87c58fb","Type":"ContainerStarted","Data":"4a24d7c9a585eef1c122a9370915983ac6d2e5ce082c2af9bee41c5695afcc1c"} Jul 22 16:41:25.491370 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:25.491340 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bc66d" event={"ID":"fb95755e-21d5-435c-9b16-eeb89295e8d0","Type":"ContainerStarted","Data":"34a0ebde2715fe6942bd789a8fafc276c649928aa6a2ce4294cc989e34417e0b"} Jul 22 16:41:25.709285 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:41:25.709227 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146e6ed1_00ed_473d_b3ef_9f50e261b7bf.slice/crio-484c4964e69023496b2eec94c301651598b391f987b607643704f00ca7e0ac3c WatchSource:0}: Error finding container 484c4964e69023496b2eec94c301651598b391f987b607643704f00ca7e0ac3c: Status 404 returned error can't find the container with id 484c4964e69023496b2eec94c301651598b391f987b607643704f00ca7e0ac3c Jul 22 16:41:26.500509 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:26.500438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zx8w2" event={"ID":"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38","Type":"ContainerStarted","Data":"8df2c7c38534b1fc04c2b9d75b82f7c389fe6abdff108be9e50562f0bdcb8d48"} Jul 22 16:41:26.500509 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:26.500513 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zx8w2" event={"ID":"659ef35b-7c38-49f0-bd4a-a8c99e6f2f38","Type":"ContainerStarted","Data":"2f4b7b7ef6132a6f475378cf3fb4028ff257ade82f7a99e965cde19447617d2c"} Jul 22 16:41:26.503031 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:26.502997 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xl265" event={"ID":"146e6ed1-00ed-473d-b3ef-9f50e261b7bf","Type":"ContainerStarted","Data":"484c4964e69023496b2eec94c301651598b391f987b607643704f00ca7e0ac3c"} Jul 22 16:41:26.526093 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:26.525658 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zx8w2" podStartSLOduration=67.036132087 podStartE2EDuration="1m8.525635041s" totalImagesPullingTime="1.489502954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:26.523603883 +0000 UTC m=+68.880000203" watchObservedRunningTime="2026-07-22 16:41:26.525635041 +0000 UTC m=+68.882031360" Jul 22 16:41:27.528365 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.528327 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk"] Jul 22 16:41:27.550885 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.550493 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv"] Jul 22 16:41:27.550885 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.550610 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk" Jul 22 16:41:27.558505 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.558411 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 16:41:27.558647 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.558595 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-tktbd\"" Jul 22 16:41:27.560220 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.558958 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 16:41:27.560220 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.559154 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 22 16:41:27.560220 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.559291 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 16:41:27.564213 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.564140 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv"] Jul 22 16:41:27.564469 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.564391 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:27.566563 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.566541 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 22 16:41:27.586355 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.586250 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk"] Jul 22 16:41:27.586355 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.586292 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv"] Jul 22 16:41:27.586355 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.586322 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv"] Jul 22 16:41:27.586587 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.586367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.589002 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.588980 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 22 16:41:27.589564 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.589541 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 22 16:41:27.589665 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.589619 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 22 16:41:27.591264 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.591243 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 22 16:41:27.629819 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.629779 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6645b5458d-ghrb4"] Jul 22 16:41:27.651261 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.651223 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.651630 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.651593 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6645b5458d-ghrb4"] Jul 22 16:41:27.654879 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.654487 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 16:41:27.654879 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.654631 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 16:41:27.654879 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.654665 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gtxm5\"" Jul 22 16:41:27.655324 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.655303 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 16:41:27.664603 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.664581 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 16:41:27.681170 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.681066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvmll\" (UniqueName: \"kubernetes.io/projected/7ec874c1-727f-4d57-bc1b-5318b6a48bae-kube-api-access-cvmll\") pod \"managed-serviceaccount-addon-agent-6dccb8f666-9dlfk\" (UID: \"7ec874c1-727f-4d57-bc1b-5318b6a48bae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk" Jul 22 16:41:27.681170 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.681106 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61b9ab5-2b0d-4e2e-8872-9673510e0251-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.681170 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.681136 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c61b9ab5-2b0d-4e2e-8872-9673510e0251-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.681840 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.681224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c61b9ab5-2b0d-4e2e-8872-9673510e0251-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.681840 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.681258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9b804989-e62f-4c72-81c2-5edd79ed2db3-klusterlet-config\") pod \"klusterlet-addon-workmgr-5894bcc7f4-2vcvv\" (UID: \"9b804989-e62f-4c72-81c2-5edd79ed2db3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:27.681965 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.681903 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxv5\" (UniqueName: \"kubernetes.io/projected/9b804989-e62f-4c72-81c2-5edd79ed2db3-kube-api-access-bdxv5\") pod \"klusterlet-addon-workmgr-5894bcc7f4-2vcvv\" (UID: \"9b804989-e62f-4c72-81c2-5edd79ed2db3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:27.681965 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.681940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7ec874c1-727f-4d57-bc1b-5318b6a48bae-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6dccb8f666-9dlfk\" (UID: \"7ec874c1-727f-4d57-bc1b-5318b6a48bae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk" Jul 22 16:41:27.682067 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.681966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c61b9ab5-2b0d-4e2e-8872-9673510e0251-ca\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.682067 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.681989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c61b9ab5-2b0d-4e2e-8872-9673510e0251-hub\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.682067 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.682016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblf4\" (UniqueName: \"kubernetes.io/projected/c61b9ab5-2b0d-4e2e-8872-9673510e0251-kube-api-access-cblf4\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.682067 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.682038 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b804989-e62f-4c72-81c2-5edd79ed2db3-tmp\") pod \"klusterlet-addon-workmgr-5894bcc7f4-2vcvv\" (UID: \"9b804989-e62f-4c72-81c2-5edd79ed2db3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:27.783049 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.782983 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b804989-e62f-4c72-81c2-5edd79ed2db3-tmp\") pod \"klusterlet-addon-workmgr-5894bcc7f4-2vcvv\" (UID: \"9b804989-e62f-4c72-81c2-5edd79ed2db3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:27.783049 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783022 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblf4\" (UniqueName: \"kubernetes.io/projected/c61b9ab5-2b0d-4e2e-8872-9673510e0251-kube-api-access-cblf4\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.783274 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61b9ab5-2b0d-4e2e-8872-9673510e0251-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.783274 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvmll\" (UniqueName: \"kubernetes.io/projected/7ec874c1-727f-4d57-bc1b-5318b6a48bae-kube-api-access-cvmll\") pod \"managed-serviceaccount-addon-agent-6dccb8f666-9dlfk\" (UID: \"7ec874c1-727f-4d57-bc1b-5318b6a48bae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk" Jul 22 16:41:27.783274 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c61b9ab5-2b0d-4e2e-8872-9673510e0251-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.783428 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-ca-trust-extracted\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.783428 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783412 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b804989-e62f-4c72-81c2-5edd79ed2db3-tmp\") pod \"klusterlet-addon-workmgr-5894bcc7f4-2vcvv\" (UID: \"9b804989-e62f-4c72-81c2-5edd79ed2db3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:27.783541 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783433 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-registry-tls\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.783541 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-registry-certificates\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.783541 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783534 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-bound-sa-token\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.783730 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7ec874c1-727f-4d57-bc1b-5318b6a48bae-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6dccb8f666-9dlfk\" (UID: \"7ec874c1-727f-4d57-bc1b-5318b6a48bae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk" Jul 22 16:41:27.783730 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c61b9ab5-2b0d-4e2e-8872-9673510e0251-hub\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.783730 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783666 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-installation-pull-secrets\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.783730 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c61b9ab5-2b0d-4e2e-8872-9673510e0251-ca\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.783963 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c61b9ab5-2b0d-4e2e-8872-9673510e0251-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.783963 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-image-registry-private-configuration\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.783963 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-trusted-ca\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.783963 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9b804989-e62f-4c72-81c2-5edd79ed2db3-klusterlet-config\") pod \"klusterlet-addon-workmgr-5894bcc7f4-2vcvv\" (UID: \"9b804989-e62f-4c72-81c2-5edd79ed2db3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:27.783963 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxv5\" (UniqueName: \"kubernetes.io/projected/9b804989-e62f-4c72-81c2-5edd79ed2db3-kube-api-access-bdxv5\") pod \"klusterlet-addon-workmgr-5894bcc7f4-2vcvv\" (UID: \"9b804989-e62f-4c72-81c2-5edd79ed2db3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:27.783963 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783873 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61b9ab5-2b0d-4e2e-8872-9673510e0251-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.783963 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.783893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vzqs\" (UniqueName: \"kubernetes.io/projected/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-kube-api-access-7vzqs\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.786264 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.786221 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c61b9ab5-2b0d-4e2e-8872-9673510e0251-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.787057 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.786929 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c61b9ab5-2b0d-4e2e-8872-9673510e0251-hub\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.787057 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.787001 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c61b9ab5-2b0d-4e2e-8872-9673510e0251-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.787225 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.787163 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c61b9ab5-2b0d-4e2e-8872-9673510e0251-ca\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.787225 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.787213 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7ec874c1-727f-4d57-bc1b-5318b6a48bae-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6dccb8f666-9dlfk\" (UID: \"7ec874c1-727f-4d57-bc1b-5318b6a48bae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk" Jul 22 16:41:27.787376 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.787354 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9b804989-e62f-4c72-81c2-5edd79ed2db3-klusterlet-config\") pod \"klusterlet-addon-workmgr-5894bcc7f4-2vcvv\" (UID: \"9b804989-e62f-4c72-81c2-5edd79ed2db3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:27.791810 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.791780 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblf4\" (UniqueName: \"kubernetes.io/projected/c61b9ab5-2b0d-4e2e-8872-9673510e0251-kube-api-access-cblf4\") pod \"cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv\" (UID: \"c61b9ab5-2b0d-4e2e-8872-9673510e0251\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.791918 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.791825 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvmll\" (UniqueName: \"kubernetes.io/projected/7ec874c1-727f-4d57-bc1b-5318b6a48bae-kube-api-access-cvmll\") pod \"managed-serviceaccount-addon-agent-6dccb8f666-9dlfk\" (UID: \"7ec874c1-727f-4d57-bc1b-5318b6a48bae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk" Jul 22 16:41:27.792150 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.792135 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxv5\" (UniqueName: \"kubernetes.io/projected/9b804989-e62f-4c72-81c2-5edd79ed2db3-kube-api-access-bdxv5\") pod \"klusterlet-addon-workmgr-5894bcc7f4-2vcvv\" (UID: \"9b804989-e62f-4c72-81c2-5edd79ed2db3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:27.872139 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.872102 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk" Jul 22 16:41:27.883228 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.883197 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:27.884483 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.884457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-ca-trust-extracted\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.884598 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.884497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-registry-tls\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.884598 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.884524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-registry-certificates\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.884598 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.884548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-bound-sa-token\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.884598 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.884571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-installation-pull-secrets\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.884807 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.884606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-image-registry-private-configuration\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.884807 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.884627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-trusted-ca\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.884807 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.884648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vzqs\" (UniqueName: \"kubernetes.io/projected/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-kube-api-access-7vzqs\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.884944 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.884891 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-ca-trust-extracted\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.885554 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.885504 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-registry-certificates\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.885948 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.885897 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-trusted-ca\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.887576 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.887555 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-registry-tls\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.887676 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.887633 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-image-registry-private-configuration\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.888443 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.888422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-installation-pull-secrets\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.897209 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.897189 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" Jul 22 16:41:27.899434 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.899411 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-bound-sa-token\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.901045 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.901002 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vzqs\" (UniqueName: \"kubernetes.io/projected/ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7-kube-api-access-7vzqs\") pod \"image-registry-6645b5458d-ghrb4\" (UID: \"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7\") " pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:27.971634 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:27.971610 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:31.995810 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:31.995661 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6645b5458d-ghrb4"] Jul 22 16:41:32.002120 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.002049 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk"] Jul 22 16:41:32.011156 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:41:32.011118 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ec874c1_727f_4d57_bc1b_5318b6a48bae.slice/crio-73bff4f7d18df53b3052cc959da3ce81ff72839a5fe0d8ac3b54a5469f46c2a2 WatchSource:0}: Error finding container 73bff4f7d18df53b3052cc959da3ce81ff72839a5fe0d8ac3b54a5469f46c2a2: Status 404 returned error can't find the container with id 73bff4f7d18df53b3052cc959da3ce81ff72839a5fe0d8ac3b54a5469f46c2a2 Jul 22 16:41:32.021895 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.021866 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv"] Jul 22 16:41:32.031643 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:41:32.031588 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b804989_e62f_4c72_81c2_5edd79ed2db3.slice/crio-693c5f171159fc6a5793233d79fa13c6810975d0b63a6885a8b7046f742ce8ae WatchSource:0}: Error finding container 693c5f171159fc6a5793233d79fa13c6810975d0b63a6885a8b7046f742ce8ae: Status 404 returned error can't find the container with id 693c5f171159fc6a5793233d79fa13c6810975d0b63a6885a8b7046f742ce8ae Jul 22 16:41:32.056670 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.056094 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv"] Jul 22 16:41:32.060280 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:41:32.059944 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc61b9ab5_2b0d_4e2e_8872_9673510e0251.slice/crio-a4c52659dc20e11ee5cd2628a25a51acf6bd3885d65417da1a54f3dd7108cdbc WatchSource:0}: Error finding container a4c52659dc20e11ee5cd2628a25a51acf6bd3885d65417da1a54f3dd7108cdbc: Status 404 returned error can't find the container with id a4c52659dc20e11ee5cd2628a25a51acf6bd3885d65417da1a54f3dd7108cdbc Jul 22 16:41:32.526858 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.526820 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" event={"ID":"c61b9ab5-2b0d-4e2e-8872-9673510e0251","Type":"ContainerStarted","Data":"a4c52659dc20e11ee5cd2628a25a51acf6bd3885d65417da1a54f3dd7108cdbc"} Jul 22 16:41:32.528205 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.528177 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" event={"ID":"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7","Type":"ContainerStarted","Data":"464ad31f8822f286b5c8dc54cbfc55b932c5a2f57080375ad445370dc6412015"} Jul 22 16:41:32.528351 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.528226 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" event={"ID":"ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7","Type":"ContainerStarted","Data":"45b80819632e2cc2d0471029ff2b38bcfc429257a79310075f765fd58e23d948"} Jul 22 16:41:32.528351 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.528306 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:41:32.529564 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.529543 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mvtsj" event={"ID":"3d12a195-9286-4276-b484-9b5ea87c58fb","Type":"ContainerStarted","Data":"cc2c2be9aa1e346962e2184416474051c90514150d56a1690a5e61b7f14c2190"} Jul 22 16:41:32.531076 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.531051 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xl265" event={"ID":"146e6ed1-00ed-473d-b3ef-9f50e261b7bf","Type":"ContainerStarted","Data":"b3208e056d1524f8e4126dfaa18f9036f7caf7ba6ba37daddf5a4b18d1524e4f"} Jul 22 16:41:32.531150 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.531099 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xl265" event={"ID":"146e6ed1-00ed-473d-b3ef-9f50e261b7bf","Type":"ContainerStarted","Data":"5c77ec417cf96d336f57e6ffeb5ea84e61cdb7ad992b71d6d7efd34ae7cb17ba"} Jul 22 16:41:32.531150 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.531124 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xl265" Jul 22 16:41:32.532941 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.532920 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bc66d" event={"ID":"fb95755e-21d5-435c-9b16-eeb89295e8d0","Type":"ContainerStarted","Data":"84f97a63570c938371f0e88d84092056cd4ccd9912e1a2ceadffd18736fbba17"} Jul 22 16:41:32.534235 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.534217 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bdf57" event={"ID":"5109af8a-ceac-4234-b0df-553ed7215c66","Type":"ContainerStarted","Data":"697ee93384dda87888248926ab9758d3042d6be6507a26d227842ff98a8504cf"} Jul 22 16:41:32.534318 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.534260 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:41:32.535238 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.535218 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" event={"ID":"9b804989-e62f-4c72-81c2-5edd79ed2db3","Type":"ContainerStarted","Data":"693c5f171159fc6a5793233d79fa13c6810975d0b63a6885a8b7046f742ce8ae"} Jul 22 16:41:32.536108 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.536090 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk" event={"ID":"7ec874c1-727f-4d57-bc1b-5318b6a48bae","Type":"ContainerStarted","Data":"73bff4f7d18df53b3052cc959da3ce81ff72839a5fe0d8ac3b54a5469f46c2a2"} Jul 22 16:41:32.548413 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.548372 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" podStartSLOduration=5.548359859 podStartE2EDuration="5.548359859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:32.547174477 +0000 UTC m=+74.903570819" watchObservedRunningTime="2026-07-22 16:41:32.548359859 +0000 UTC m=+74.904756154" Jul 22 16:41:32.565695 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.565657 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bc66d" podStartSLOduration=17.736470965 podStartE2EDuration="33.565645836s" totalImagesPullingTime="15.829174871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:32.565156113 +0000 UTC m=+74.921552446" watchObservedRunningTime="2026-07-22 16:41:32.565645836 +0000 UTC m=+74.922042154" Jul 22 16:41:32.582434 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.582394 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mvtsj" podStartSLOduration=67.312409323 podStartE2EDuration="1m14.582381818s" totalImagesPullingTime="7.269972495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:32.58205281 +0000 UTC m=+74.938449128" watchObservedRunningTime="2026-07-22 16:41:32.582381818 +0000 UTC m=+74.938778114" Jul 22 16:41:32.601301 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.601264 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bdf57" podStartSLOduration=67.348438196 podStartE2EDuration="1m14.601251924s" totalImagesPullingTime="7.252813728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:32.600788171 +0000 UTC m=+74.957184493" watchObservedRunningTime="2026-07-22 16:41:32.601251924 +0000 UTC m=+74.957648241" Jul 22 16:41:32.623131 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:32.623096 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xl265" podStartSLOduration=34.54107602 podStartE2EDuration="40.623076121s" totalImagesPullingTime="6.082000101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:32.622441043 +0000 UTC m=+74.978837362" watchObservedRunningTime="2026-07-22 16:41:32.623076121 +0000 UTC m=+74.979472436" Jul 22 16:41:36.556062 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:36.556017 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" event={"ID":"9b804989-e62f-4c72-81c2-5edd79ed2db3","Type":"ContainerStarted","Data":"95c0b23243beb7e69631b0748d285a79db63c6692ae8d6feb895309308dcb105"} Jul 22 16:41:36.556532 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:36.556229 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:36.557998 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:36.557972 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk" event={"ID":"7ec874c1-727f-4d57-bc1b-5318b6a48bae","Type":"ContainerStarted","Data":"9b301cae5b30c86234c760c52c2a9fb77bfdb32fa2c2a13bb9cdbb6a3944738f"} Jul 22 16:41:36.558121 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:36.558053 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" Jul 22 16:41:36.572284 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:36.572233 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5894bcc7f4-2vcvv" podStartSLOduration=5.5330746699999995 podStartE2EDuration="9.572215462s" totalImagesPullingTime="4.039140792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:36.570472635 +0000 UTC m=+78.926868952" watchObservedRunningTime="2026-07-22 16:41:36.572215462 +0000 UTC m=+78.928611761" Jul 22 16:41:36.598295 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:36.598201 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dccb8f666-9dlfk" podStartSLOduration=5.598721308 podStartE2EDuration="9.59818242s" totalImagesPullingTime="3.999461112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:36.59809761 +0000 UTC m=+78.954493930" watchObservedRunningTime="2026-07-22 16:41:36.59818242 +0000 UTC m=+78.954578740" Jul 22 16:41:39.572346 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:39.572312 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" event={"ID":"c61b9ab5-2b0d-4e2e-8872-9673510e0251","Type":"ContainerStarted","Data":"c64fcaa0b84b5c07a58f0c5aaa5e3a7b5c5b1f2e7a369f5a5226e17666429862"} Jul 22 16:41:39.572858 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:39.572363 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" event={"ID":"c61b9ab5-2b0d-4e2e-8872-9673510e0251","Type":"ContainerStarted","Data":"767356bba7ffcbd4a16a7ad4a4e243d3b8f0657a5778e9aa571515b1d79b0337"} Jul 22 16:41:39.572858 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:39.572382 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" event={"ID":"c61b9ab5-2b0d-4e2e-8872-9673510e0251","Type":"ContainerStarted","Data":"54cfad18aacdfcd627a63211f1161f5ef7c13286bf1dbf256c3eadd3f160de39"} Jul 22 16:41:39.594494 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:39.594443 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dc8fc99cc-j4tnv" podStartSLOduration=5.951076197 podStartE2EDuration="12.594426723s" totalImagesPullingTime="6.643350526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:39.593362275 +0000 UTC m=+81.949758593" watchObservedRunningTime="2026-07-22 16:41:39.594426723 +0000 UTC m=+81.950823040" Jul 22 16:41:42.543646 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:42.543535 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xl265" Jul 22 16:41:42.924426 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:42.924390 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rfsnz"] Jul 22 16:41:42.931877 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:42.931849 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:42.934542 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:42.934514 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 16:41:42.934769 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:42.934547 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 16:41:42.934835 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:42.934627 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 16:41:42.935247 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:42.935229 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 16:41:42.935247 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:42.935241 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qv2jv\"" Jul 22 16:41:42.935457 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:42.935440 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 16:41:42.935566 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:42.935519 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 16:41:43.004160 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.004106 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-879nv\" (UniqueName: \"kubernetes.io/projected/592621a7-e5b4-4f71-8641-20161ab4115c-kube-api-access-879nv\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.004160 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.004159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/592621a7-e5b4-4f71-8641-20161ab4115c-metrics-client-ca\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.004403 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.004211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/592621a7-e5b4-4f71-8641-20161ab4115c-root\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.004403 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.004240 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-textfile\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.004403 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.004280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.004403 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.004338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-tls\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.004403 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.004377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.004632 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.004438 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-wtmp\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.004632 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.004474 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/592621a7-e5b4-4f71-8641-20161ab4115c-sys\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.105522 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.105484 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-wtmp\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.105522 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.105523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/592621a7-e5b4-4f71-8641-20161ab4115c-sys\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.105775 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.105542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-879nv\" (UniqueName: \"kubernetes.io/projected/592621a7-e5b4-4f71-8641-20161ab4115c-kube-api-access-879nv\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.105775 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.105557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/592621a7-e5b4-4f71-8641-20161ab4115c-metrics-client-ca\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.105775 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.105595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/592621a7-e5b4-4f71-8641-20161ab4115c-root\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.105775 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.105620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-textfile\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.105775 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.105628 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/592621a7-e5b4-4f71-8641-20161ab4115c-sys\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.105775 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.105644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.105775 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.105696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-tls\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.105775 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.105711 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/592621a7-e5b4-4f71-8641-20161ab4115c-root\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.105775 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.105731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.106217 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.105835 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-wtmp\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.106217 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:43.105968 2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 16:41:43.106217 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:41:43.106048 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-tls podName:592621a7-e5b4-4f71-8641-20161ab4115c nodeName:}" failed. No retries permitted until 2026-07-22 16:41:43.606026375 +0000 UTC m=+85.962422678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-tls") pod "node-exporter-rfsnz" (UID: "592621a7-e5b4-4f71-8641-20161ab4115c") : secret "node-exporter-tls" not found Jul 22 16:41:43.106379 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.106239 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/592621a7-e5b4-4f71-8641-20161ab4115c-metrics-client-ca\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.106379 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.106329 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.106507 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.106486 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-textfile\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.108359 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.108341 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.113771 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.113749 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-879nv\" (UniqueName: \"kubernetes.io/projected/592621a7-e5b4-4f71-8641-20161ab4115c-kube-api-access-879nv\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.610625 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.610591 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-tls\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.613300 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.613272 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/592621a7-e5b4-4f71-8641-20161ab4115c-node-exporter-tls\") pod \"node-exporter-rfsnz\" (UID: \"592621a7-e5b4-4f71-8641-20161ab4115c\") " pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.843479 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:43.843443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rfsnz" Jul 22 16:41:43.855681 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:41:43.855637 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592621a7_e5b4_4f71_8641_20161ab4115c.slice/crio-a69e017a2e6cf86794cfea9fe8655312d01ba56b6b8b382dc47c75f31bfa038a WatchSource:0}: Error finding container a69e017a2e6cf86794cfea9fe8655312d01ba56b6b8b382dc47c75f31bfa038a: Status 404 returned error can't find the container with id a69e017a2e6cf86794cfea9fe8655312d01ba56b6b8b382dc47c75f31bfa038a Jul 22 16:41:44.589932 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:44.589887 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfsnz" event={"ID":"592621a7-e5b4-4f71-8641-20161ab4115c","Type":"ContainerStarted","Data":"a69e017a2e6cf86794cfea9fe8655312d01ba56b6b8b382dc47c75f31bfa038a"} Jul 22 16:41:45.594606 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:45.594572 2577 generic.go:356] "Generic (PLEG): container finished" podID="592621a7-e5b4-4f71-8641-20161ab4115c" containerID="9016d3970c0365923e7c006ac8d43ba51edf4364e003c01752f2c4cb30a2cc1f" exitCode=0 Jul 22 16:41:45.595027 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:45.594625 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfsnz" event={"ID":"592621a7-e5b4-4f71-8641-20161ab4115c","Type":"ContainerDied","Data":"9016d3970c0365923e7c006ac8d43ba51edf4364e003c01752f2c4cb30a2cc1f"} Jul 22 16:41:46.600385 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:46.600350 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfsnz" event={"ID":"592621a7-e5b4-4f71-8641-20161ab4115c","Type":"ContainerStarted","Data":"007fe21ef7ce806662afb824f94f791c246f62baf8d1770d987aa1eabdc37c7b"} Jul 22 16:41:46.600894 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:46.600408 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfsnz" event={"ID":"592621a7-e5b4-4f71-8641-20161ab4115c","Type":"ContainerStarted","Data":"e911b0225048ea96bffaea518f8fe261a68b6de8b46f1a9ad4de432734492918"} Jul 22 16:41:46.627457 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:46.627406 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rfsnz" podStartSLOduration=3.738663399 podStartE2EDuration="4.627391346s" totalImagesPullingTime="825.444633ms" totalInitContainerRuntime="63.283314ms" isStatefulPod=true podCreationTimestamp="2026-07-22 16:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:46.62689045 +0000 UTC m=+88.983286792" watchObservedRunningTime="2026-07-22 16:41:46.627391346 +0000 UTC m=+88.983787664" Jul 22 16:41:53.546307 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:41:53.546276 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6645b5458d-ghrb4" Jul 22 16:42:03.544613 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:42:03.544584 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bdf57" Jul 22 16:45:18.050290 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:45:18.050262 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 16:46:56.089567 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.089533 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j"] Jul 22 16:46:56.091538 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.091521 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.096595 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.096566 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 16:46:56.096775 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.096725 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 16:46:56.097166 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.097150 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 16:46:56.098420 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.098403 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4qzcs\"" Jul 22 16:46:56.099239 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.099225 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 16:46:56.122685 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.122659 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j"] Jul 22 16:46:56.216458 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.216433 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/06516966-2f01-4e52-8c83-65c2a860ab9c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.216607 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.216472 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06516966-2f01-4e52-8c83-65c2a860ab9c-tmp\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.216607 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.216545 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/06516966-2f01-4e52-8c83-65c2a860ab9c-manifests\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.216607 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.216573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06516966-2f01-4e52-8c83-65c2a860ab9c-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.216607 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.216596 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzqxj\" (UniqueName: \"kubernetes.io/projected/06516966-2f01-4e52-8c83-65c2a860ab9c-kube-api-access-dzqxj\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.317370 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.317331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/06516966-2f01-4e52-8c83-65c2a860ab9c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.317551 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.317384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06516966-2f01-4e52-8c83-65c2a860ab9c-tmp\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.317551 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.317433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/06516966-2f01-4e52-8c83-65c2a860ab9c-manifests\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.317551 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.317454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06516966-2f01-4e52-8c83-65c2a860ab9c-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.317551 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.317478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzqxj\" (UniqueName: \"kubernetes.io/projected/06516966-2f01-4e52-8c83-65c2a860ab9c-kube-api-access-dzqxj\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.317875 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.317851 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/06516966-2f01-4e52-8c83-65c2a860ab9c-manifests\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.319765 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.319721 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06516966-2f01-4e52-8c83-65c2a860ab9c-tmp\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.319991 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.319958 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06516966-2f01-4e52-8c83-65c2a860ab9c-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.320047 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.320033 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/06516966-2f01-4e52-8c83-65c2a860ab9c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.353921 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.353849 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzqxj\" (UniqueName: \"kubernetes.io/projected/06516966-2f01-4e52-8c83-65c2a860ab9c-kube-api-access-dzqxj\") pod \"opendatahub-operator-controller-manager-55b7bf7664-clb4j\" (UID: \"06516966-2f01-4e52-8c83-65c2a860ab9c\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.401776 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.401718 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:46:56.565487 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:46:56.565457 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06516966_2f01_4e52_8c83_65c2a860ab9c.slice/crio-27743a211ac689b76d66ffa457b034346f96b43cd7695ab5617d600435889465 WatchSource:0}: Error finding container 27743a211ac689b76d66ffa457b034346f96b43cd7695ab5617d600435889465: Status 404 returned error can't find the container with id 27743a211ac689b76d66ffa457b034346f96b43cd7695ab5617d600435889465 Jul 22 16:46:56.567582 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.567560 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:46:56.568646 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.568626 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j"] Jul 22 16:46:56.599776 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:56.599749 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" event={"ID":"06516966-2f01-4e52-8c83-65c2a860ab9c","Type":"ContainerStarted","Data":"27743a211ac689b76d66ffa457b034346f96b43cd7695ab5617d600435889465"} Jul 22 16:46:59.612459 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:46:59.612424 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" event={"ID":"06516966-2f01-4e52-8c83-65c2a860ab9c","Type":"ContainerStarted","Data":"56aae79947e9f65980ec9640f4a92649ed443511e02c7fb8189700f86d183321"} Jul 22 16:47:00.617480 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:00.617446 2577 generic.go:356] "Generic (PLEG): container finished" podID="06516966-2f01-4e52-8c83-65c2a860ab9c" containerID="56aae79947e9f65980ec9640f4a92649ed443511e02c7fb8189700f86d183321" exitCode=0 Jul 22 16:47:00.617823 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:00.617527 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" event={"ID":"06516966-2f01-4e52-8c83-65c2a860ab9c","Type":"ContainerDied","Data":"56aae79947e9f65980ec9640f4a92649ed443511e02c7fb8189700f86d183321"} Jul 22 16:47:01.623084 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:01.623048 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" event={"ID":"06516966-2f01-4e52-8c83-65c2a860ab9c","Type":"ContainerStarted","Data":"f934de00dbeb20ad9810533a6830e4e86c049a08764613afa007721b068c788e"} Jul 22 16:47:01.623431 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:01.623200 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:47:01.842904 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:01.842850 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" podStartSLOduration=1.549240883 podStartE2EDuration="5.842832923s" totalImagesPullingTime="3.213787861s" totalInitContainerRuntime="1.079804179s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:46:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:47:01.842556083 +0000 UTC m=+404.198952402" watchObservedRunningTime="2026-07-22 16:47:01.842832923 +0000 UTC m=+404.199229241" Jul 22 16:47:12.629706 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:12.629677 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-clb4j" Jul 22 16:47:18.015543 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.015509 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-c467c957f-7btr8"] Jul 22 16:47:18.021787 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.021770 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" Jul 22 16:47:18.033832 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.033809 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 16:47:18.033959 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.033834 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 16:47:18.033959 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.033811 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jgl2w\"" Jul 22 16:47:18.033959 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.033950 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 16:47:18.034107 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.033952 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 16:47:18.084891 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.084858 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c467c957f-7btr8"] Jul 22 16:47:18.180080 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.180050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14769731-231c-4538-8692-265628d70b78-tmp\") pod \"kube-auth-proxy-c467c957f-7btr8\" (UID: \"14769731-231c-4538-8692-265628d70b78\") " pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" Jul 22 16:47:18.180080 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.180082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgj4\" (UniqueName: \"kubernetes.io/projected/14769731-231c-4538-8692-265628d70b78-kube-api-access-9tgj4\") pod \"kube-auth-proxy-c467c957f-7btr8\" (UID: \"14769731-231c-4538-8692-265628d70b78\") " pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" Jul 22 16:47:18.180293 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.180114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14769731-231c-4538-8692-265628d70b78-tls-certs\") pod \"kube-auth-proxy-c467c957f-7btr8\" (UID: \"14769731-231c-4538-8692-265628d70b78\") " pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" Jul 22 16:47:18.280910 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.280819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14769731-231c-4538-8692-265628d70b78-tmp\") pod \"kube-auth-proxy-c467c957f-7btr8\" (UID: \"14769731-231c-4538-8692-265628d70b78\") " pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" Jul 22 16:47:18.280910 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.280868 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgj4\" (UniqueName: \"kubernetes.io/projected/14769731-231c-4538-8692-265628d70b78-kube-api-access-9tgj4\") pod \"kube-auth-proxy-c467c957f-7btr8\" (UID: \"14769731-231c-4538-8692-265628d70b78\") " pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" Jul 22 16:47:18.281118 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.280946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14769731-231c-4538-8692-265628d70b78-tls-certs\") pod \"kube-auth-proxy-c467c957f-7btr8\" (UID: \"14769731-231c-4538-8692-265628d70b78\") " pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" Jul 22 16:47:18.284082 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.284051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14769731-231c-4538-8692-265628d70b78-tls-certs\") pod \"kube-auth-proxy-c467c957f-7btr8\" (UID: \"14769731-231c-4538-8692-265628d70b78\") " pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" Jul 22 16:47:18.284286 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.284268 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14769731-231c-4538-8692-265628d70b78-tmp\") pod \"kube-auth-proxy-c467c957f-7btr8\" (UID: \"14769731-231c-4538-8692-265628d70b78\") " pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" Jul 22 16:47:18.328167 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.328133 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgj4\" (UniqueName: \"kubernetes.io/projected/14769731-231c-4538-8692-265628d70b78-kube-api-access-9tgj4\") pod \"kube-auth-proxy-c467c957f-7btr8\" (UID: \"14769731-231c-4538-8692-265628d70b78\") " pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" Jul 22 16:47:18.329890 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.329874 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" Jul 22 16:47:18.605468 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.605387 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c467c957f-7btr8"] Jul 22 16:47:18.611687 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:47:18.611655 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14769731_231c_4538_8692_265628d70b78.slice/crio-927ded1847eff20a94f1ae2c226b9cc2d4a44fb7e28c65df2e44e155b59837b4 WatchSource:0}: Error finding container 927ded1847eff20a94f1ae2c226b9cc2d4a44fb7e28c65df2e44e155b59837b4: Status 404 returned error can't find the container with id 927ded1847eff20a94f1ae2c226b9cc2d4a44fb7e28c65df2e44e155b59837b4 Jul 22 16:47:18.682604 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:18.682562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" event={"ID":"14769731-231c-4538-8692-265628d70b78","Type":"ContainerStarted","Data":"927ded1847eff20a94f1ae2c226b9cc2d4a44fb7e28c65df2e44e155b59837b4"} Jul 22 16:47:21.695581 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:21.695547 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" event={"ID":"14769731-231c-4538-8692-265628d70b78","Type":"ContainerStarted","Data":"005f194b8dad69921b6c6908260fbc9b58ae954a64210eef07c4ae78b338fbe2"} Jul 22 16:47:21.767029 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:21.766979 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-c467c957f-7btr8" podStartSLOduration=2.127324396 podStartE2EDuration="4.766962758s" totalImagesPullingTime="2.639638362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:47:21.766501488 +0000 UTC m=+424.122897800" watchObservedRunningTime="2026-07-22 16:47:21.766962758 +0000 UTC m=+424.123359074" Jul 22 16:47:22.326931 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.326900 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz"] Jul 22 16:47:22.329624 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.329607 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.342600 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:47:22.342575 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"istio-kubeconfig\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" Jul 22 16:47:22.342802 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.342789 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rthqn\"" Jul 22 16:47:22.356019 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.355997 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 16:47:22.356151 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.355997 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 16:47:22.356151 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.355997 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 16:47:22.418363 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.418336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.418490 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.418375 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psj9f\" (UniqueName: \"kubernetes.io/projected/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-kube-api-access-psj9f\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.418490 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.418412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-token\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.418490 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.418426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-cacerts\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.418490 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.418441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.418490 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.418460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.418650 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.418537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-local-certs\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.424486 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.424459 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz"] Jul 22 16:47:22.519484 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.519453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-token\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.519484 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.519485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-cacerts\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.519719 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.519502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.519719 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.519520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.519719 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.519667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-local-certs\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.519719 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.519699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.519970 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.519769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psj9f\" (UniqueName: \"kubernetes.io/projected/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-kube-api-access-psj9f\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.520290 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.520261 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.521885 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.521859 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-local-certs\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.521964 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.521872 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-cacerts\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.522191 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.522168 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.543085 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.543063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-token\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:22.546242 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:22.546216 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psj9f\" (UniqueName: \"kubernetes.io/projected/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-kube-api-access-psj9f\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:23.520344 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:47:23.520313 2577 secret.go:189] Couldn't get secret openshift-ingress/istio-kubeconfig: failed to sync secret cache: timed out waiting for the condition Jul 22 16:47:23.520772 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:47:23.520400 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-kubeconfig podName:5fce331c-1c07-4560-a1f4-d7ffcdb295a3 nodeName:}" failed. No retries permitted until 2026-07-22 16:47:24.020383125 +0000 UTC m=+426.376779424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "istio-kubeconfig" (UniqueName: "kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-kubeconfig") pod "istiod-openshift-gateway-7574459656-zz9zz" (UID: "5fce331c-1c07-4560-a1f4-d7ffcdb295a3") : failed to sync secret cache: timed out waiting for the condition Jul 22 16:47:23.670455 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:23.670425 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 16:47:24.031753 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:24.031717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:24.034019 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:24.033990 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zz9zz\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:24.138254 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:24.138222 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:24.315969 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:24.315866 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz"] Jul 22 16:47:24.318428 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:47:24.318399 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fce331c_1c07_4560_a1f4_d7ffcdb295a3.slice/crio-ebaacb4e6a81c2a9d3bf0e5e28f13f4204b001a50eb85bc6137a285933f0182f WatchSource:0}: Error finding container ebaacb4e6a81c2a9d3bf0e5e28f13f4204b001a50eb85bc6137a285933f0182f: Status 404 returned error can't find the container with id ebaacb4e6a81c2a9d3bf0e5e28f13f4204b001a50eb85bc6137a285933f0182f Jul 22 16:47:24.707899 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:24.707868 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" event={"ID":"5fce331c-1c07-4560-a1f4-d7ffcdb295a3","Type":"ContainerStarted","Data":"ebaacb4e6a81c2a9d3bf0e5e28f13f4204b001a50eb85bc6137a285933f0182f"} Jul 22 16:47:26.751116 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:26.751065 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:47:26.751389 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:26.751143 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:47:27.724119 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:27.724086 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" event={"ID":"5fce331c-1c07-4560-a1f4-d7ffcdb295a3","Type":"ContainerStarted","Data":"59eaa50e34b40a291da2b896368ec8c8c1e013735675a71f8b64920997ef172e"} Jul 22 16:47:27.724357 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:27.724327 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:27.726163 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:27.726134 2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-zz9zz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 16:47:27.726307 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:27.726187 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" podUID="5fce331c-1c07-4560-a1f4-d7ffcdb295a3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 16:47:27.796966 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:27.796910 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" podStartSLOduration=3.366392893 podStartE2EDuration="5.79689499s" totalImagesPullingTime="2.430502097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:47:27.795788767 +0000 UTC m=+430.152185104" watchObservedRunningTime="2026-07-22 16:47:27.79689499 +0000 UTC m=+430.153291306" Jul 22 16:47:28.728526 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:28.728494 2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-zz9zz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 16:47:28.728693 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:28.728552 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" podUID="5fce331c-1c07-4560-a1f4-d7ffcdb295a3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 16:47:30.227307 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.227276 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp"] Jul 22 16:47:30.230426 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.230410 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.232927 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.232904 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-cfvqg\"" Jul 22 16:47:30.280921 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.280893 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp"] Jul 22 16:47:30.383222 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.383196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.383222 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.383227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.383393 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.383250 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.383393 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.383270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7p7m\" (UniqueName: \"kubernetes.io/projected/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-kube-api-access-z7p7m\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.383393 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.383333 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.383393 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.383357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.383393 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.383375 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.383552 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.383398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.383552 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.383412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.483717 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.483648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7p7m\" (UniqueName: \"kubernetes.io/projected/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-kube-api-access-z7p7m\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.483717 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.483691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.483717 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.483712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.483966 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.483732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.483966 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.483788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.483966 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.483809 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.483966 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.483845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.483966 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.483870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.483966 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.483904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.484284 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.484171 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.484284 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.484221 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.484284 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.484271 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.484430 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.484328 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.484609 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.484588 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.485944 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.485922 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.486312 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.486297 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.524419 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.524391 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7p7m\" (UniqueName: \"kubernetes.io/projected/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-kube-api-access-z7p7m\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.545525 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.545505 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dqhhlp\" (UID: \"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:30.840200 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:30.840130 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:31.008639 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:31.008609 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp"] Jul 22 16:47:31.011404 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:47:31.011376 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9ceec5_7144_49f6_8cf7_9cb2ec07a0f7.slice/crio-41f5a5598cc5b4c78104cc14afb8f5c923521e260dee9685dc100054c487a226 WatchSource:0}: Error finding container 41f5a5598cc5b4c78104cc14afb8f5c923521e260dee9685dc100054c487a226: Status 404 returned error can't find the container with id 41f5a5598cc5b4c78104cc14afb8f5c923521e260dee9685dc100054c487a226 Jul 22 16:47:31.729047 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:31.729012 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:31.742927 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:31.742893 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" event={"ID":"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7","Type":"ContainerStarted","Data":"41f5a5598cc5b4c78104cc14afb8f5c923521e260dee9685dc100054c487a226"} Jul 22 16:47:32.987610 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:32.987573 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz"] Jul 22 16:47:32.988020 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:32.987852 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" podUID="5fce331c-1c07-4560-a1f4-d7ffcdb295a3" containerName="discovery" containerID="cri-o://59eaa50e34b40a291da2b896368ec8c8c1e013735675a71f8b64920997ef172e" gracePeriod=30 Jul 22 16:47:33.546201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.546167 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p"] Jul 22 16:47:33.552615 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.552593 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" Jul 22 16:47:33.567799 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.567779 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dthwn\"" Jul 22 16:47:33.568632 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.568601 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 22 16:47:33.593082 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.593058 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm"] Jul 22 16:47:33.597992 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.597946 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" Jul 22 16:47:33.601025 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.601003 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 22 16:47:33.601598 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.601576 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-npjx9\"" Jul 22 16:47:33.626425 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.626402 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p"] Jul 22 16:47:33.679284 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.679258 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm"] Jul 22 16:47:33.709826 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.709796 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a4ea855-dfad-4558-94bd-090f17c49657-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p4drm\" (UID: \"7a4ea855-dfad-4558-94bd-090f17c49657\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" Jul 22 16:47:33.709826 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.709844 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c00ea5e5-2cb3-44a2-96b8-110b00d55b63-cert\") pod \"kserve-controller-manager-d6f4b65b8-c7l5p\" (UID: \"c00ea5e5-2cb3-44a2-96b8-110b00d55b63\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" Jul 22 16:47:33.710094 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.709879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxw9r\" (UniqueName: \"kubernetes.io/projected/c00ea5e5-2cb3-44a2-96b8-110b00d55b63-kube-api-access-rxw9r\") pod \"kserve-controller-manager-d6f4b65b8-c7l5p\" (UID: \"c00ea5e5-2cb3-44a2-96b8-110b00d55b63\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" Jul 22 16:47:33.710094 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.710051 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjhvm\" (UniqueName: \"kubernetes.io/projected/7a4ea855-dfad-4558-94bd-090f17c49657-kube-api-access-mjhvm\") pod \"llmisvc-controller-manager-777c6bd7d5-p4drm\" (UID: \"7a4ea855-dfad-4558-94bd-090f17c49657\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" Jul 22 16:47:33.716073 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.716043 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:47:33.716203 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.716102 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:47:33.716203 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.716128 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:47:33.761125 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.761096 2577 generic.go:356] "Generic (PLEG): container finished" podID="5fce331c-1c07-4560-a1f4-d7ffcdb295a3" containerID="59eaa50e34b40a291da2b896368ec8c8c1e013735675a71f8b64920997ef172e" exitCode=0 Jul 22 16:47:33.761245 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.761208 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" event={"ID":"5fce331c-1c07-4560-a1f4-d7ffcdb295a3","Type":"ContainerDied","Data":"59eaa50e34b40a291da2b896368ec8c8c1e013735675a71f8b64920997ef172e"} Jul 22 16:47:33.809476 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.809452 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:33.810414 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.810393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a4ea855-dfad-4558-94bd-090f17c49657-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p4drm\" (UID: \"7a4ea855-dfad-4558-94bd-090f17c49657\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" Jul 22 16:47:33.810414 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.810421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c00ea5e5-2cb3-44a2-96b8-110b00d55b63-cert\") pod \"kserve-controller-manager-d6f4b65b8-c7l5p\" (UID: \"c00ea5e5-2cb3-44a2-96b8-110b00d55b63\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" Jul 22 16:47:33.810587 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.810440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxw9r\" (UniqueName: \"kubernetes.io/projected/c00ea5e5-2cb3-44a2-96b8-110b00d55b63-kube-api-access-rxw9r\") pod \"kserve-controller-manager-d6f4b65b8-c7l5p\" (UID: \"c00ea5e5-2cb3-44a2-96b8-110b00d55b63\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" Jul 22 16:47:33.810587 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.810483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjhvm\" (UniqueName: \"kubernetes.io/projected/7a4ea855-dfad-4558-94bd-090f17c49657-kube-api-access-mjhvm\") pod \"llmisvc-controller-manager-777c6bd7d5-p4drm\" (UID: \"7a4ea855-dfad-4558-94bd-090f17c49657\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" Jul 22 16:47:33.812630 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.812609 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a4ea855-dfad-4558-94bd-090f17c49657-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p4drm\" (UID: \"7a4ea855-dfad-4558-94bd-090f17c49657\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" Jul 22 16:47:33.812846 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.812825 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c00ea5e5-2cb3-44a2-96b8-110b00d55b63-cert\") pod \"kserve-controller-manager-d6f4b65b8-c7l5p\" (UID: \"c00ea5e5-2cb3-44a2-96b8-110b00d55b63\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" Jul 22 16:47:33.832125 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.832094 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjhvm\" (UniqueName: \"kubernetes.io/projected/7a4ea855-dfad-4558-94bd-090f17c49657-kube-api-access-mjhvm\") pod \"llmisvc-controller-manager-777c6bd7d5-p4drm\" (UID: \"7a4ea855-dfad-4558-94bd-090f17c49657\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" Jul 22 16:47:33.833014 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.832994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxw9r\" (UniqueName: \"kubernetes.io/projected/c00ea5e5-2cb3-44a2-96b8-110b00d55b63-kube-api-access-rxw9r\") pod \"kserve-controller-manager-d6f4b65b8-c7l5p\" (UID: \"c00ea5e5-2cb3-44a2-96b8-110b00d55b63\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" Jul 22 16:47:33.863887 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.863858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" Jul 22 16:47:33.909574 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.909540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" Jul 22 16:47:33.911795 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.911774 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-cacerts\") pod \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " Jul 22 16:47:33.911915 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.911831 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-csr-dns-cert\") pod \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " Jul 22 16:47:33.911915 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.911862 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-csr-ca-configmap\") pod \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " Jul 22 16:47:33.911915 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.911888 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-local-certs\") pod \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " Jul 22 16:47:33.912068 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.911917 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-kubeconfig\") pod \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " Jul 22 16:47:33.912068 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.911946 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psj9f\" (UniqueName: \"kubernetes.io/projected/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-kube-api-access-psj9f\") pod \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " Jul 22 16:47:33.912068 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.911992 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-token\") pod \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\" (UID: \"5fce331c-1c07-4560-a1f4-d7ffcdb295a3\") " Jul 22 16:47:33.912819 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.912534 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5fce331c-1c07-4560-a1f4-d7ffcdb295a3" (UID: "5fce331c-1c07-4560-a1f4-d7ffcdb295a3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:47:33.915332 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.915288 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "5fce331c-1c07-4560-a1f4-d7ffcdb295a3" (UID: "5fce331c-1c07-4560-a1f4-d7ffcdb295a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:47:33.915332 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.915294 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5fce331c-1c07-4560-a1f4-d7ffcdb295a3" (UID: "5fce331c-1c07-4560-a1f4-d7ffcdb295a3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:47:33.915704 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.915682 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5fce331c-1c07-4560-a1f4-d7ffcdb295a3" (UID: "5fce331c-1c07-4560-a1f4-d7ffcdb295a3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:47:33.916049 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.916016 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-cacerts" (OuterVolumeSpecName: "cacerts") pod "5fce331c-1c07-4560-a1f4-d7ffcdb295a3" (UID: "5fce331c-1c07-4560-a1f4-d7ffcdb295a3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:47:33.916489 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.916467 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-kube-api-access-psj9f" (OuterVolumeSpecName: "kube-api-access-psj9f") pod "5fce331c-1c07-4560-a1f4-d7ffcdb295a3" (UID: "5fce331c-1c07-4560-a1f4-d7ffcdb295a3"). InnerVolumeSpecName "kube-api-access-psj9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:47:33.916669 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:33.916651 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-local-certs" (OuterVolumeSpecName: "local-certs") pod "5fce331c-1c07-4560-a1f4-d7ffcdb295a3" (UID: "5fce331c-1c07-4560-a1f4-d7ffcdb295a3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:47:34.013465 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.013432 2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-cacerts\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:47:34.013465 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.013467 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-csr-dns-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:47:34.013831 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.013484 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-csr-ca-configmap\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:47:34.013831 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.013498 2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-local-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:47:34.013831 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.013511 2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-kubeconfig\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:47:34.013831 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.013524 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psj9f\" (UniqueName: \"kubernetes.io/projected/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-kube-api-access-psj9f\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:47:34.013831 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.013536 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fce331c-1c07-4560-a1f4-d7ffcdb295a3-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:47:34.054488 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.054416 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p"] Jul 22 16:47:34.057619 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:47:34.057587 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00ea5e5_2cb3_44a2_96b8_110b00d55b63.slice/crio-cb9762053c8f345e036305d00e61b704ba18675192a01cec3093d2e5418f6dde WatchSource:0}: Error finding container cb9762053c8f345e036305d00e61b704ba18675192a01cec3093d2e5418f6dde: Status 404 returned error can't find the container with id cb9762053c8f345e036305d00e61b704ba18675192a01cec3093d2e5418f6dde Jul 22 16:47:34.114547 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.114251 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm"] Jul 22 16:47:34.116581 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:47:34.116550 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7a4ea855_dfad_4558_94bd_090f17c49657.slice/crio-09a86f73e4a8d62a86a0e26efb1e9ddba793f20d7c714236a19b674556e45373 WatchSource:0}: Error finding container 09a86f73e4a8d62a86a0e26efb1e9ddba793f20d7c714236a19b674556e45373: Status 404 returned error can't find the container with id 09a86f73e4a8d62a86a0e26efb1e9ddba793f20d7c714236a19b674556e45373 Jul 22 16:47:34.767784 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.767723 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" event={"ID":"c00ea5e5-2cb3-44a2-96b8-110b00d55b63","Type":"ContainerStarted","Data":"cb9762053c8f345e036305d00e61b704ba18675192a01cec3093d2e5418f6dde"} Jul 22 16:47:34.769142 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.769116 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" Jul 22 16:47:34.769277 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.769115 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz" event={"ID":"5fce331c-1c07-4560-a1f4-d7ffcdb295a3","Type":"ContainerDied","Data":"ebaacb4e6a81c2a9d3bf0e5e28f13f4204b001a50eb85bc6137a285933f0182f"} Jul 22 16:47:34.769277 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.769269 2577 scope.go:117] "RemoveContainer" containerID="59eaa50e34b40a291da2b896368ec8c8c1e013735675a71f8b64920997ef172e" Jul 22 16:47:34.771012 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.770976 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" event={"ID":"8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7","Type":"ContainerStarted","Data":"33e98132ed94db7f5afda605e72643fb145ea157b5d920922bb1e95055d9c05d"} Jul 22 16:47:34.772788 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.772756 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" event={"ID":"7a4ea855-dfad-4558-94bd-090f17c49657","Type":"ContainerStarted","Data":"09a86f73e4a8d62a86a0e26efb1e9ddba793f20d7c714236a19b674556e45373"} Jul 22 16:47:34.840491 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.840453 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:34.845394 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.845369 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:34.847676 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.847650 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz"] Jul 22 16:47:34.907247 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:34.907209 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zz9zz"] Jul 22 16:47:35.022213 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:35.022112 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" podStartSLOduration=2.319635948 podStartE2EDuration="5.022087973s" totalImagesPullingTime="2.702452025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:47:35.008205341 +0000 UTC m=+437.364601719" watchObservedRunningTime="2026-07-22 16:47:35.022087973 +0000 UTC m=+437.378484291" Jul 22 16:47:35.779550 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:35.779519 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:35.780732 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:35.780702 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dqhhlp" Jul 22 16:47:36.192137 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:36.192098 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fce331c-1c07-4560-a1f4-d7ffcdb295a3" path="/var/lib/kubelet/pods/5fce331c-1c07-4560-a1f4-d7ffcdb295a3/volumes" Jul 22 16:47:36.785877 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:36.785825 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" event={"ID":"c00ea5e5-2cb3-44a2-96b8-110b00d55b63","Type":"ContainerStarted","Data":"d054a3064211524e0690cced9ed47d9c96c347375c00a5b3938f03cd179fce74"} Jul 22 16:47:36.786057 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:36.785965 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" Jul 22 16:47:36.840691 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:36.840594 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" podStartSLOduration=1.331176942 podStartE2EDuration="3.840577147s" totalImagesPullingTime="2.509400205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:47:36.838273713 +0000 UTC m=+439.194670070" watchObservedRunningTime="2026-07-22 16:47:36.840577147 +0000 UTC m=+439.196973540" Jul 22 16:47:37.791531 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:37.791495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" event={"ID":"7a4ea855-dfad-4558-94bd-090f17c49657","Type":"ContainerStarted","Data":"c114aba8ec6f323b3594d053d9f37cc9b082b49553db806491d579bf17dacb05"} Jul 22 16:47:37.791909 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:37.791558 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" Jul 22 16:47:37.850987 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:47:37.850939 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" podStartSLOduration=1.769418664 podStartE2EDuration="4.85092316s" totalImagesPullingTime="3.081504496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:47:37.84823742 +0000 UTC m=+440.204633736" watchObservedRunningTime="2026-07-22 16:47:37.85092316 +0000 UTC m=+440.207319477" Jul 22 16:48:07.796637 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:07.796603 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c7l5p" Jul 22 16:48:08.798098 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:08.798067 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p4drm" Jul 22 16:48:18.441513 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.441478 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkbjp"] Jul 22 16:48:18.441885 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.441794 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fce331c-1c07-4560-a1f4-d7ffcdb295a3" containerName="discovery" Jul 22 16:48:18.441885 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.441807 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fce331c-1c07-4560-a1f4-d7ffcdb295a3" containerName="discovery" Jul 22 16:48:18.441885 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.441865 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fce331c-1c07-4560-a1f4-d7ffcdb295a3" containerName="discovery" Jul 22 16:48:18.444756 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.444713 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" Jul 22 16:48:18.447896 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.447872 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 16:48:18.448690 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.448672 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 16:48:18.448828 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.448674 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-cscj4\"" Jul 22 16:48:18.456069 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.456046 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkbjp"] Jul 22 16:48:18.536051 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.536025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwf85\" (UniqueName: \"kubernetes.io/projected/b783b7ce-9734-4901-993a-35b4bd58c964-kube-api-access-hwf85\") pod \"kuadrant-operator-catalog-gkbjp\" (UID: \"b783b7ce-9734-4901-993a-35b4bd58c964\") " pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" Jul 22 16:48:18.636662 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.636626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwf85\" (UniqueName: \"kubernetes.io/projected/b783b7ce-9734-4901-993a-35b4bd58c964-kube-api-access-hwf85\") pod \"kuadrant-operator-catalog-gkbjp\" (UID: \"b783b7ce-9734-4901-993a-35b4bd58c964\") " pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" Jul 22 16:48:18.645494 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.645466 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwf85\" (UniqueName: \"kubernetes.io/projected/b783b7ce-9734-4901-993a-35b4bd58c964-kube-api-access-hwf85\") pod \"kuadrant-operator-catalog-gkbjp\" (UID: \"b783b7ce-9734-4901-993a-35b4bd58c964\") " pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" Jul 22 16:48:18.754327 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.754261 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" Jul 22 16:48:18.806546 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.806477 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkbjp"] Jul 22 16:48:18.880401 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.880379 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkbjp"] Jul 22 16:48:18.882915 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:48:18.882892 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb783b7ce_9734_4901_993a_35b4bd58c964.slice/crio-6c7b8c73072ccb90f7244fe665e4207dd6135031645c91063aebe4e6b0ee3a54 WatchSource:0}: Error finding container 6c7b8c73072ccb90f7244fe665e4207dd6135031645c91063aebe4e6b0ee3a54: Status 404 returned error can't find the container with id 6c7b8c73072ccb90f7244fe665e4207dd6135031645c91063aebe4e6b0ee3a54 Jul 22 16:48:18.948947 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:18.948920 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" event={"ID":"b783b7ce-9734-4901-993a-35b4bd58c964","Type":"ContainerStarted","Data":"6c7b8c73072ccb90f7244fe665e4207dd6135031645c91063aebe4e6b0ee3a54"} Jul 22 16:48:19.015862 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:19.015803 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z29kx"] Jul 22 16:48:19.018715 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:19.018701 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-z29kx" Jul 22 16:48:19.027630 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:19.027606 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z29kx"] Jul 22 16:48:19.140242 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:19.140216 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-579kk\" (UniqueName: \"kubernetes.io/projected/afe65379-c439-437c-be70-c07424eb758b-kube-api-access-579kk\") pod \"kuadrant-operator-catalog-z29kx\" (UID: \"afe65379-c439-437c-be70-c07424eb758b\") " pod="kuadrant-system/kuadrant-operator-catalog-z29kx" Jul 22 16:48:19.241472 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:19.241447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579kk\" (UniqueName: \"kubernetes.io/projected/afe65379-c439-437c-be70-c07424eb758b-kube-api-access-579kk\") pod \"kuadrant-operator-catalog-z29kx\" (UID: \"afe65379-c439-437c-be70-c07424eb758b\") " pod="kuadrant-system/kuadrant-operator-catalog-z29kx" Jul 22 16:48:19.251231 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:19.251212 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-579kk\" (UniqueName: \"kubernetes.io/projected/afe65379-c439-437c-be70-c07424eb758b-kube-api-access-579kk\") pod \"kuadrant-operator-catalog-z29kx\" (UID: \"afe65379-c439-437c-be70-c07424eb758b\") " pod="kuadrant-system/kuadrant-operator-catalog-z29kx" Jul 22 16:48:19.327754 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:19.327675 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-z29kx" Jul 22 16:48:19.457477 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:19.457447 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z29kx"] Jul 22 16:48:19.460343 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:48:19.460320 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe65379_c439_437c_be70_c07424eb758b.slice/crio-4b17ce824657cb362f2232d2af1c269f7754269cf191bd91a47fa6d291095093 WatchSource:0}: Error finding container 4b17ce824657cb362f2232d2af1c269f7754269cf191bd91a47fa6d291095093: Status 404 returned error can't find the container with id 4b17ce824657cb362f2232d2af1c269f7754269cf191bd91a47fa6d291095093 Jul 22 16:48:19.954527 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:19.954494 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-z29kx" event={"ID":"afe65379-c439-437c-be70-c07424eb758b","Type":"ContainerStarted","Data":"4b17ce824657cb362f2232d2af1c269f7754269cf191bd91a47fa6d291095093"} Jul 22 16:48:21.966127 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:21.966090 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-z29kx" event={"ID":"afe65379-c439-437c-be70-c07424eb758b","Type":"ContainerStarted","Data":"8bb1aaa0c59ce527d891da2367706247339f33086072242ce0dfdac6b7412747"} Jul 22 16:48:21.967596 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:21.967564 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" event={"ID":"b783b7ce-9734-4901-993a-35b4bd58c964","Type":"ContainerStarted","Data":"8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3"} Jul 22 16:48:21.967699 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:21.967632 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" podUID="b783b7ce-9734-4901-993a-35b4bd58c964" containerName="registry-server" containerID="cri-o://8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3" gracePeriod=2 Jul 22 16:48:21.981613 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:21.981572 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-z29kx" podStartSLOduration=1.146004467 podStartE2EDuration="2.981558261s" totalImagesPullingTime="1.835553794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:48:21.980087743 +0000 UTC m=+484.336484071" watchObservedRunningTime="2026-07-22 16:48:21.981558261 +0000 UTC m=+484.337954578" Jul 22 16:48:21.995235 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:21.995200 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" podStartSLOduration=1.5854228959999999 podStartE2EDuration="3.995187377s" totalImagesPullingTime="2.409764481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:48:21.994357924 +0000 UTC m=+484.350754240" watchObservedRunningTime="2026-07-22 16:48:21.995187377 +0000 UTC m=+484.351583694" Jul 22 16:48:22.200436 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:22.200414 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" Jul 22 16:48:22.268510 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:22.268443 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwf85\" (UniqueName: \"kubernetes.io/projected/b783b7ce-9734-4901-993a-35b4bd58c964-kube-api-access-hwf85\") pod \"b783b7ce-9734-4901-993a-35b4bd58c964\" (UID: \"b783b7ce-9734-4901-993a-35b4bd58c964\") " Jul 22 16:48:22.270448 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:22.270424 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b783b7ce-9734-4901-993a-35b4bd58c964-kube-api-access-hwf85" (OuterVolumeSpecName: "kube-api-access-hwf85") pod "b783b7ce-9734-4901-993a-35b4bd58c964" (UID: "b783b7ce-9734-4901-993a-35b4bd58c964"). InnerVolumeSpecName "kube-api-access-hwf85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:48:22.369656 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:22.369632 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwf85\" (UniqueName: \"kubernetes.io/projected/b783b7ce-9734-4901-993a-35b4bd58c964-kube-api-access-hwf85\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:48:22.973140 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:22.973105 2577 generic.go:356] "Generic (PLEG): container finished" podID="b783b7ce-9734-4901-993a-35b4bd58c964" containerID="8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3" exitCode=0 Jul 22 16:48:22.973601 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:22.973162 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" Jul 22 16:48:22.973601 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:22.973191 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" event={"ID":"b783b7ce-9734-4901-993a-35b4bd58c964","Type":"ContainerDied","Data":"8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3"} Jul 22 16:48:22.973601 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:22.973236 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gkbjp" event={"ID":"b783b7ce-9734-4901-993a-35b4bd58c964","Type":"ContainerDied","Data":"6c7b8c73072ccb90f7244fe665e4207dd6135031645c91063aebe4e6b0ee3a54"} Jul 22 16:48:22.973601 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:22.973255 2577 scope.go:117] "RemoveContainer" containerID="8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3" Jul 22 16:48:22.981772 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:22.981751 2577 scope.go:117] "RemoveContainer" containerID="8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3" Jul 22 16:48:22.982037 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:48:22.982016 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3\": container with ID starting with 8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3 not found: ID does not exist" containerID="8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3" Jul 22 16:48:22.982098 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:22.982047 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3"} err="failed to get container status \"8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3\": rpc error: code = NotFound desc = could not find container \"8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3\": container with ID starting with 8f38f91185686f7ed139e479d3ae5a70cccade36d4865b20565e3ef688496dc3 not found: ID does not exist" Jul 22 16:48:22.995919 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:22.995894 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkbjp"] Jul 22 16:48:23.004946 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:23.004926 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkbjp"] Jul 22 16:48:24.185801 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:24.185770 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b783b7ce-9734-4901-993a-35b4bd58c964" path="/var/lib/kubelet/pods/b783b7ce-9734-4901-993a-35b4bd58c964/volumes" Jul 22 16:48:29.328343 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:29.328310 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-z29kx" Jul 22 16:48:29.328781 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:29.328400 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-z29kx" Jul 22 16:48:29.349758 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:29.349702 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-z29kx" Jul 22 16:48:30.027227 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:30.027197 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-z29kx" Jul 22 16:48:50.319128 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.319088 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx"] Jul 22 16:48:50.319588 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.319477 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b783b7ce-9734-4901-993a-35b4bd58c964" containerName="registry-server" Jul 22 16:48:50.319588 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.319495 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b783b7ce-9734-4901-993a-35b4bd58c964" containerName="registry-server" Jul 22 16:48:50.319588 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.319565 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b783b7ce-9734-4901-993a-35b4bd58c964" containerName="registry-server" Jul 22 16:48:50.322315 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.322300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx" Jul 22 16:48:50.326238 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.326219 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 22 16:48:50.326485 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.326471 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-rj9cx\"" Jul 22 16:48:50.341453 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.341430 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx"] Jul 22 16:48:50.484921 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.484893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjp8l\" (UniqueName: \"kubernetes.io/projected/3287f248-c04f-4e37-a86b-edb3c10715e0-kube-api-access-xjp8l\") pod \"dns-operator-controller-manager-864dd78c4c-d2qkx\" (UID: \"3287f248-c04f-4e37-a86b-edb3c10715e0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx" Jul 22 16:48:50.586340 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.586252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjp8l\" (UniqueName: \"kubernetes.io/projected/3287f248-c04f-4e37-a86b-edb3c10715e0-kube-api-access-xjp8l\") pod \"dns-operator-controller-manager-864dd78c4c-d2qkx\" (UID: \"3287f248-c04f-4e37-a86b-edb3c10715e0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx" Jul 22 16:48:50.597511 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.597485 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjp8l\" (UniqueName: \"kubernetes.io/projected/3287f248-c04f-4e37-a86b-edb3c10715e0-kube-api-access-xjp8l\") pod \"dns-operator-controller-manager-864dd78c4c-d2qkx\" (UID: \"3287f248-c04f-4e37-a86b-edb3c10715e0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx" Jul 22 16:48:50.632279 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.632251 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx" Jul 22 16:48:50.759205 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:50.759035 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx"] Jul 22 16:48:50.761801 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:48:50.761768 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3287f248_c04f_4e37_a86b_edb3c10715e0.slice/crio-44e1fbda6c7863e4b30dbb5aabc68301fb0c883fe742fd47e294173aadaf270e WatchSource:0}: Error finding container 44e1fbda6c7863e4b30dbb5aabc68301fb0c883fe742fd47e294173aadaf270e: Status 404 returned error can't find the container with id 44e1fbda6c7863e4b30dbb5aabc68301fb0c883fe742fd47e294173aadaf270e Jul 22 16:48:51.090808 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:51.090773 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx" event={"ID":"3287f248-c04f-4e37-a86b-edb3c10715e0","Type":"ContainerStarted","Data":"44e1fbda6c7863e4b30dbb5aabc68301fb0c883fe742fd47e294173aadaf270e"} Jul 22 16:48:53.099426 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:53.099394 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx" event={"ID":"3287f248-c04f-4e37-a86b-edb3c10715e0","Type":"ContainerStarted","Data":"97f68e590fe80c76e2cbb797725d9e1c4680421f67c90cbfe10c95149a2cc2ae"} Jul 22 16:48:53.099877 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:53.099507 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx" Jul 22 16:48:53.117437 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:53.117393 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx" podStartSLOduration=0.993501147 podStartE2EDuration="3.117380821s" totalImagesPullingTime="2.123879674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:48:53.116347129 +0000 UTC m=+515.472743445" watchObservedRunningTime="2026-07-22 16:48:53.117380821 +0000 UTC m=+515.473777137" Jul 22 16:48:53.735504 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:53.735473 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9"] Jul 22 16:48:53.738766 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:53.738729 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:48:53.740985 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:53.740964 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xzlnq\"" Jul 22 16:48:53.749604 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:53.749582 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9"] Jul 22 16:48:53.915137 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:53.915102 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2cf\" (UniqueName: \"kubernetes.io/projected/1016ee7a-0667-457e-bde0-0ad4c3f0e62c-kube-api-access-6c2cf\") pod \"limitador-operator-controller-manager-859d7b78bd-4jtc9\" (UID: \"1016ee7a-0667-457e-bde0-0ad4c3f0e62c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:48:54.016565 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:54.016481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2cf\" (UniqueName: \"kubernetes.io/projected/1016ee7a-0667-457e-bde0-0ad4c3f0e62c-kube-api-access-6c2cf\") pod \"limitador-operator-controller-manager-859d7b78bd-4jtc9\" (UID: \"1016ee7a-0667-457e-bde0-0ad4c3f0e62c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:48:54.024202 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:54.024176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2cf\" (UniqueName: \"kubernetes.io/projected/1016ee7a-0667-457e-bde0-0ad4c3f0e62c-kube-api-access-6c2cf\") pod \"limitador-operator-controller-manager-859d7b78bd-4jtc9\" (UID: \"1016ee7a-0667-457e-bde0-0ad4c3f0e62c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:48:54.048475 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:54.048446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:48:54.183223 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:48:54.183194 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1016ee7a_0667_457e_bde0_0ad4c3f0e62c.slice/crio-00ad76dbf067c9fd56787cedd6f817d6e5db4a2b2a1b53a15bf238ec4b3ae270 WatchSource:0}: Error finding container 00ad76dbf067c9fd56787cedd6f817d6e5db4a2b2a1b53a15bf238ec4b3ae270: Status 404 returned error can't find the container with id 00ad76dbf067c9fd56787cedd6f817d6e5db4a2b2a1b53a15bf238ec4b3ae270 Jul 22 16:48:54.186964 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:54.186944 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9"] Jul 22 16:48:55.114004 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:55.113962 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" event={"ID":"1016ee7a-0667-457e-bde0-0ad4c3f0e62c","Type":"ContainerStarted","Data":"00ad76dbf067c9fd56787cedd6f817d6e5db4a2b2a1b53a15bf238ec4b3ae270"} Jul 22 16:48:56.120201 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:56.120122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" event={"ID":"1016ee7a-0667-457e-bde0-0ad4c3f0e62c","Type":"ContainerStarted","Data":"1f84ae3144ba4bc32ff8a8170945296bf39148a92abe7fb75c4c7c4ee748845f"} Jul 22 16:48:56.120556 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:56.120252 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:48:56.137955 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:56.137914 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" podStartSLOduration=1.517615307 podStartE2EDuration="3.137899512s" totalImagesPullingTime="1.620284205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:48:56.137173794 +0000 UTC m=+518.493570112" watchObservedRunningTime="2026-07-22 16:48:56.137899512 +0000 UTC m=+518.494295899" Jul 22 16:48:58.821585 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:58.821555 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn"] Jul 22 16:48:58.824874 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:58.824858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:48:58.828510 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:58.828488 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ctrlr\"" Jul 22 16:48:58.845334 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:58.845311 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn"] Jul 22 16:48:58.958023 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:58.957989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2644e9d-f022-4395-bba4-49b1ec88cb90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" (UID: \"a2644e9d-f022-4395-bba4-49b1ec88cb90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:48:58.958173 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:58.958039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9w9b\" (UniqueName: \"kubernetes.io/projected/a2644e9d-f022-4395-bba4-49b1ec88cb90-kube-api-access-j9w9b\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" (UID: \"a2644e9d-f022-4395-bba4-49b1ec88cb90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:48:59.058404 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:59.058370 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9w9b\" (UniqueName: \"kubernetes.io/projected/a2644e9d-f022-4395-bba4-49b1ec88cb90-kube-api-access-j9w9b\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" (UID: \"a2644e9d-f022-4395-bba4-49b1ec88cb90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:48:59.058549 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:59.058435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2644e9d-f022-4395-bba4-49b1ec88cb90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" (UID: \"a2644e9d-f022-4395-bba4-49b1ec88cb90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:48:59.058782 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:59.058765 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2644e9d-f022-4395-bba4-49b1ec88cb90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" (UID: \"a2644e9d-f022-4395-bba4-49b1ec88cb90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:48:59.070117 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:59.070095 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9w9b\" (UniqueName: \"kubernetes.io/projected/a2644e9d-f022-4395-bba4-49b1ec88cb90-kube-api-access-j9w9b\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" (UID: \"a2644e9d-f022-4395-bba4-49b1ec88cb90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:48:59.134910 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:59.134854 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:48:59.299303 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:48:59.299269 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn"] Jul 22 16:48:59.302411 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:48:59.302382 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2644e9d_f022_4395_bba4_49b1ec88cb90.slice/crio-47d00fb330475e79cd275295cd65130c4781fc077b1b41a03a16c3e1a1951c9e WatchSource:0}: Error finding container 47d00fb330475e79cd275295cd65130c4781fc077b1b41a03a16c3e1a1951c9e: Status 404 returned error can't find the container with id 47d00fb330475e79cd275295cd65130c4781fc077b1b41a03a16c3e1a1951c9e Jul 22 16:49:00.138172 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:00.138132 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" event={"ID":"a2644e9d-f022-4395-bba4-49b1ec88cb90","Type":"ContainerStarted","Data":"47d00fb330475e79cd275295cd65130c4781fc077b1b41a03a16c3e1a1951c9e"} Jul 22 16:49:02.059928 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:02.059893 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-p248h"] Jul 22 16:49:02.063110 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:02.063091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p248h" Jul 22 16:49:02.066626 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:02.066609 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xfz57\"" Jul 22 16:49:02.076973 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:02.076948 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-p248h"] Jul 22 16:49:02.100701 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:02.100675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-587qm\" (UniqueName: \"kubernetes.io/projected/a62547fc-ed8d-432d-950c-1be19941c257-kube-api-access-587qm\") pod \"authorino-operator-75cc8d5cb7-p248h\" (UID: \"a62547fc-ed8d-432d-950c-1be19941c257\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-p248h" Jul 22 16:49:02.201649 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:02.201586 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-587qm\" (UniqueName: \"kubernetes.io/projected/a62547fc-ed8d-432d-950c-1be19941c257-kube-api-access-587qm\") pod \"authorino-operator-75cc8d5cb7-p248h\" (UID: \"a62547fc-ed8d-432d-950c-1be19941c257\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-p248h" Jul 22 16:49:02.217714 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:02.217688 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-587qm\" (UniqueName: \"kubernetes.io/projected/a62547fc-ed8d-432d-950c-1be19941c257-kube-api-access-587qm\") pod \"authorino-operator-75cc8d5cb7-p248h\" (UID: \"a62547fc-ed8d-432d-950c-1be19941c257\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-p248h" Jul 22 16:49:02.373147 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:02.373060 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p248h" Jul 22 16:49:02.512658 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:02.512627 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-p248h"] Jul 22 16:49:02.516910 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:49:02.516879 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62547fc_ed8d_432d_950c_1be19941c257.slice/crio-91f36056e6ee2123130bc6a2704dad20bdaa34a698da45be842ffb961661904a WatchSource:0}: Error finding container 91f36056e6ee2123130bc6a2704dad20bdaa34a698da45be842ffb961661904a: Status 404 returned error can't find the container with id 91f36056e6ee2123130bc6a2704dad20bdaa34a698da45be842ffb961661904a Jul 22 16:49:03.151786 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:03.151749 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p248h" event={"ID":"a62547fc-ed8d-432d-950c-1be19941c257","Type":"ContainerStarted","Data":"91f36056e6ee2123130bc6a2704dad20bdaa34a698da45be842ffb961661904a"} Jul 22 16:49:04.110506 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:04.110477 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-d2qkx" Jul 22 16:49:06.168052 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:06.168018 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" event={"ID":"a2644e9d-f022-4395-bba4-49b1ec88cb90","Type":"ContainerStarted","Data":"48dc58fd2be7ec9e7e529adb4cf9d4e74faaa5c5b7137ef43c95619c9f77b397"} Jul 22 16:49:06.168490 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:06.168118 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:06.169444 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:06.169417 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p248h" event={"ID":"a62547fc-ed8d-432d-950c-1be19941c257","Type":"ContainerStarted","Data":"f95e3664126a4d17ac29aaecd379407bf1a8b10640d925a3a268f8e4d5b713f8"} Jul 22 16:49:06.169568 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:06.169551 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p248h" Jul 22 16:49:06.187693 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:06.187653 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" podStartSLOduration=1.7582849710000001 podStartE2EDuration="8.18764149s" totalImagesPullingTime="6.429356519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:49:06.186837624 +0000 UTC m=+528.543233941" watchObservedRunningTime="2026-07-22 16:49:06.18764149 +0000 UTC m=+528.544037805" Jul 22 16:49:06.204313 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:06.204271 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p248h" podStartSLOduration=0.95292365 podStartE2EDuration="4.204257637s" totalImagesPullingTime="3.251333987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:49:06.203311999 +0000 UTC m=+528.559708336" watchObservedRunningTime="2026-07-22 16:49:06.204257637 +0000 UTC m=+528.560653954" Jul 22 16:49:07.127084 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:07.127051 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:49:17.175086 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:17.175004 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:17.175086 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:17.175085 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p248h" Jul 22 16:49:18.889465 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.889432 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn"] Jul 22 16:49:18.889873 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.889665 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" containerName="manager" containerID="cri-o://48dc58fd2be7ec9e7e529adb4cf9d4e74faaa5c5b7137ef43c95619c9f77b397" gracePeriod=2 Jul 22 16:49:18.903715 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.903688 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn"] Jul 22 16:49:18.908699 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:18.908670 2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a2644e9d-f022-4395-bba4-49b1ec88cb90\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-r49zn\": pods \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:18.921542 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.921518 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f"] Jul 22 16:49:18.922047 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.922026 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" containerName="manager" Jul 22 16:49:18.922047 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.922045 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" containerName="manager" Jul 22 16:49:18.922193 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.922118 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" containerName="manager" Jul 22 16:49:18.925034 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.925015 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" Jul 22 16:49:18.925539 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.925490 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9"] Jul 22 16:49:18.925699 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.925668 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" podUID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" containerName="manager" containerID="cri-o://1f84ae3144ba4bc32ff8a8170945296bf39148a92abe7fb75c4c7c4ee748845f" gracePeriod=2 Jul 22 16:49:18.931640 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.931613 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9"] Jul 22 16:49:18.949340 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.949316 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f"] Jul 22 16:49:18.962108 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.962084 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667"] Jul 22 16:49:18.962544 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.962526 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" containerName="manager" Jul 22 16:49:18.962627 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.962548 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" containerName="manager" Jul 22 16:49:18.962678 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.962626 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" containerName="manager" Jul 22 16:49:18.965818 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.965796 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667" Jul 22 16:49:18.976046 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:18.976015 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:18.980147 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:18.980124 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4jtc9\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:49:18.982576 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:18.982538 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:18.985180 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:18.985155 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667"] Jul 22 16:49:19.041944 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.041915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mtj2\" (UniqueName: \"kubernetes.io/projected/6e55b967-5a94-4fa9-9d2e-b6a33d84fbc0-kube-api-access-2mtj2\") pod \"limitador-operator-controller-manager-859d7b78bd-72667\" (UID: \"6e55b967-5a94-4fa9-9d2e-b6a33d84fbc0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667" Jul 22 16:49:19.042058 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.041994 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/095d417a-ad24-43b0-9415-cb35460eefce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nx48f\" (UID: \"095d417a-ad24-43b0-9415-cb35460eefce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" Jul 22 16:49:19.042058 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.042051 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8vr\" (UniqueName: \"kubernetes.io/projected/095d417a-ad24-43b0-9415-cb35460eefce-kube-api-access-hr8vr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nx48f\" (UID: \"095d417a-ad24-43b0-9415-cb35460eefce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" Jul 22 16:49:19.143362 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.143333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/095d417a-ad24-43b0-9415-cb35460eefce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nx48f\" (UID: \"095d417a-ad24-43b0-9415-cb35460eefce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" Jul 22 16:49:19.143502 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.143407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8vr\" (UniqueName: \"kubernetes.io/projected/095d417a-ad24-43b0-9415-cb35460eefce-kube-api-access-hr8vr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nx48f\" (UID: \"095d417a-ad24-43b0-9415-cb35460eefce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" Jul 22 16:49:19.143502 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.143457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mtj2\" (UniqueName: \"kubernetes.io/projected/6e55b967-5a94-4fa9-9d2e-b6a33d84fbc0-kube-api-access-2mtj2\") pod \"limitador-operator-controller-manager-859d7b78bd-72667\" (UID: \"6e55b967-5a94-4fa9-9d2e-b6a33d84fbc0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667" Jul 22 16:49:19.143754 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.143719 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/095d417a-ad24-43b0-9415-cb35460eefce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nx48f\" (UID: \"095d417a-ad24-43b0-9415-cb35460eefce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" Jul 22 16:49:19.154286 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.154268 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8vr\" (UniqueName: \"kubernetes.io/projected/095d417a-ad24-43b0-9415-cb35460eefce-kube-api-access-hr8vr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nx48f\" (UID: \"095d417a-ad24-43b0-9415-cb35460eefce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" Jul 22 16:49:19.154575 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.154555 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mtj2\" (UniqueName: \"kubernetes.io/projected/6e55b967-5a94-4fa9-9d2e-b6a33d84fbc0-kube-api-access-2mtj2\") pod \"limitador-operator-controller-manager-859d7b78bd-72667\" (UID: \"6e55b967-5a94-4fa9-9d2e-b6a33d84fbc0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667" Jul 22 16:49:19.169327 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.169308 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:19.171220 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.171195 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4jtc9\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:49:19.171789 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.171774 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:49:19.173022 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.172993 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:19.174729 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.174710 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4jtc9\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:49:19.176369 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.176352 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:19.221833 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.221806 2577 generic.go:356] "Generic (PLEG): container finished" podID="a2644e9d-f022-4395-bba4-49b1ec88cb90" containerID="48dc58fd2be7ec9e7e529adb4cf9d4e74faaa5c5b7137ef43c95619c9f77b397" exitCode=0 Jul 22 16:49:19.221917 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.221849 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:19.221917 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.221904 2577 scope.go:117] "RemoveContainer" containerID="48dc58fd2be7ec9e7e529adb4cf9d4e74faaa5c5b7137ef43c95619c9f77b397" Jul 22 16:49:19.223180 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.223161 2577 generic.go:356] "Generic (PLEG): container finished" podID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" containerID="1f84ae3144ba4bc32ff8a8170945296bf39148a92abe7fb75c4c7c4ee748845f" exitCode=0 Jul 22 16:49:19.223271 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.223207 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:49:19.223911 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.223891 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4jtc9\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:49:19.225531 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.225511 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:19.227178 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.227154 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4jtc9\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:49:19.229699 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.229683 2577 scope.go:117] "RemoveContainer" containerID="48dc58fd2be7ec9e7e529adb4cf9d4e74faaa5c5b7137ef43c95619c9f77b397" Jul 22 16:49:19.229965 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.229947 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48dc58fd2be7ec9e7e529adb4cf9d4e74faaa5c5b7137ef43c95619c9f77b397\": container with ID starting with 48dc58fd2be7ec9e7e529adb4cf9d4e74faaa5c5b7137ef43c95619c9f77b397 not found: ID does not exist" containerID="48dc58fd2be7ec9e7e529adb4cf9d4e74faaa5c5b7137ef43c95619c9f77b397" Jul 22 16:49:19.230019 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.229973 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48dc58fd2be7ec9e7e529adb4cf9d4e74faaa5c5b7137ef43c95619c9f77b397"} err="failed to get container status \"48dc58fd2be7ec9e7e529adb4cf9d4e74faaa5c5b7137ef43c95619c9f77b397\": rpc error: code = NotFound desc = could not find container \"48dc58fd2be7ec9e7e529adb4cf9d4e74faaa5c5b7137ef43c95619c9f77b397\": container with ID starting with 48dc58fd2be7ec9e7e529adb4cf9d4e74faaa5c5b7137ef43c95619c9f77b397 not found: ID does not exist" Jul 22 16:49:19.230019 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.229987 2577 scope.go:117] "RemoveContainer" containerID="1f84ae3144ba4bc32ff8a8170945296bf39148a92abe7fb75c4c7c4ee748845f" Jul 22 16:49:19.230858 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.230838 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:19.236877 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.236857 2577 scope.go:117] "RemoveContainer" containerID="1f84ae3144ba4bc32ff8a8170945296bf39148a92abe7fb75c4c7c4ee748845f" Jul 22 16:49:19.237110 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.237088 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f84ae3144ba4bc32ff8a8170945296bf39148a92abe7fb75c4c7c4ee748845f\": container with ID starting with 1f84ae3144ba4bc32ff8a8170945296bf39148a92abe7fb75c4c7c4ee748845f not found: ID does not exist" containerID="1f84ae3144ba4bc32ff8a8170945296bf39148a92abe7fb75c4c7c4ee748845f" Jul 22 16:49:19.237188 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.237113 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f84ae3144ba4bc32ff8a8170945296bf39148a92abe7fb75c4c7c4ee748845f"} err="failed to get container status \"1f84ae3144ba4bc32ff8a8170945296bf39148a92abe7fb75c4c7c4ee748845f\": rpc error: code = NotFound desc = could not find container \"1f84ae3144ba4bc32ff8a8170945296bf39148a92abe7fb75c4c7c4ee748845f\": container with ID starting with 1f84ae3144ba4bc32ff8a8170945296bf39148a92abe7fb75c4c7c4ee748845f not found: ID does not exist" Jul 22 16:49:19.244374 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.244356 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9w9b\" (UniqueName: \"kubernetes.io/projected/a2644e9d-f022-4395-bba4-49b1ec88cb90-kube-api-access-j9w9b\") pod \"a2644e9d-f022-4395-bba4-49b1ec88cb90\" (UID: \"a2644e9d-f022-4395-bba4-49b1ec88cb90\") " Jul 22 16:49:19.244433 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.244415 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2644e9d-f022-4395-bba4-49b1ec88cb90-extensions-socket-volume\") pod \"a2644e9d-f022-4395-bba4-49b1ec88cb90\" (UID: \"a2644e9d-f022-4395-bba4-49b1ec88cb90\") " Jul 22 16:49:19.244504 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.244493 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c2cf\" (UniqueName: \"kubernetes.io/projected/1016ee7a-0667-457e-bde0-0ad4c3f0e62c-kube-api-access-6c2cf\") pod \"1016ee7a-0667-457e-bde0-0ad4c3f0e62c\" (UID: \"1016ee7a-0667-457e-bde0-0ad4c3f0e62c\") " Jul 22 16:49:19.244688 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.244670 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2644e9d-f022-4395-bba4-49b1ec88cb90-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a2644e9d-f022-4395-bba4-49b1ec88cb90" (UID: "a2644e9d-f022-4395-bba4-49b1ec88cb90"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:49:19.246283 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.246257 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1016ee7a-0667-457e-bde0-0ad4c3f0e62c-kube-api-access-6c2cf" (OuterVolumeSpecName: "kube-api-access-6c2cf") pod "1016ee7a-0667-457e-bde0-0ad4c3f0e62c" (UID: "1016ee7a-0667-457e-bde0-0ad4c3f0e62c"). InnerVolumeSpecName "kube-api-access-6c2cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:49:19.246362 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.246305 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2644e9d-f022-4395-bba4-49b1ec88cb90-kube-api-access-j9w9b" (OuterVolumeSpecName: "kube-api-access-j9w9b") pod "a2644e9d-f022-4395-bba4-49b1ec88cb90" (UID: "a2644e9d-f022-4395-bba4-49b1ec88cb90"). InnerVolumeSpecName "kube-api-access-j9w9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:49:19.320991 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.320972 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" Jul 22 16:49:19.326387 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.326366 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667" Jul 22 16:49:19.349028 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.345341 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2644e9d-f022-4395-bba4-49b1ec88cb90-extensions-socket-volume\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:49:19.349028 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.345379 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c2cf\" (UniqueName: \"kubernetes.io/projected/1016ee7a-0667-457e-bde0-0ad4c3f0e62c-kube-api-access-6c2cf\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:49:19.349028 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.345394 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9w9b\" (UniqueName: \"kubernetes.io/projected/a2644e9d-f022-4395-bba4-49b1ec88cb90-kube-api-access-j9w9b\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:49:19.463691 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.463661 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f"] Jul 22 16:49:19.467398 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:49:19.467367 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095d417a_ad24_43b0_9415_cb35460eefce.slice/crio-db23a993029874773fac03885c7d844720240f631087a956ffcc826e73d2cb6e WatchSource:0}: Error finding container db23a993029874773fac03885c7d844720240f631087a956ffcc826e73d2cb6e: Status 404 returned error can't find the container with id db23a993029874773fac03885c7d844720240f631087a956ffcc826e73d2cb6e Jul 22 16:49:19.487490 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:19.487465 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667"] Jul 22 16:49:19.501815 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:49:19.501785 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e55b967_5a94_4fa9_9d2e_b6a33d84fbc0.slice/crio-b514fd19d94a8d970af22c3f9ef2577040227f174105537c21d9fa36df9f7b15 WatchSource:0}: Error finding container b514fd19d94a8d970af22c3f9ef2577040227f174105537c21d9fa36df9f7b15: Status 404 returned error can't find the container with id b514fd19d94a8d970af22c3f9ef2577040227f174105537c21d9fa36df9f7b15 Jul 22 16:49:19.533902 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.533867 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:19.536048 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.536023 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4jtc9\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:49:19.537757 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.537712 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4jtc9\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4jtc9" Jul 22 16:49:19.541002 ip-10-0-136-75 kubenswrapper[2577]: E0722 16:49:19.540976 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-r49zn\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r49zn" Jul 22 16:49:20.185821 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:20.185725 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1016ee7a-0667-457e-bde0-0ad4c3f0e62c" path="/var/lib/kubelet/pods/1016ee7a-0667-457e-bde0-0ad4c3f0e62c/volumes" Jul 22 16:49:20.186150 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:20.186076 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2644e9d-f022-4395-bba4-49b1ec88cb90" path="/var/lib/kubelet/pods/a2644e9d-f022-4395-bba4-49b1ec88cb90/volumes" Jul 22 16:49:20.228733 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:20.228702 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667" event={"ID":"6e55b967-5a94-4fa9-9d2e-b6a33d84fbc0","Type":"ContainerStarted","Data":"74f622d5269d6b3714feefd0a3b2f1f0a02344951175a73baef12640006af279"} Jul 22 16:49:20.228888 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:20.228772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667" event={"ID":"6e55b967-5a94-4fa9-9d2e-b6a33d84fbc0","Type":"ContainerStarted","Data":"b514fd19d94a8d970af22c3f9ef2577040227f174105537c21d9fa36df9f7b15"} Jul 22 16:49:20.228888 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:20.228795 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667" Jul 22 16:49:20.231252 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:20.231229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" event={"ID":"095d417a-ad24-43b0-9415-cb35460eefce","Type":"ContainerStarted","Data":"16c66a98568849b628d679f7b53ea47703b4cb38131b6d0cdc2b856b951ab9be"} Jul 22 16:49:20.231361 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:20.231264 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" event={"ID":"095d417a-ad24-43b0-9415-cb35460eefce","Type":"ContainerStarted","Data":"db23a993029874773fac03885c7d844720240f631087a956ffcc826e73d2cb6e"} Jul 22 16:49:20.231361 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:20.231307 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" Jul 22 16:49:20.248286 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:20.248234 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667" podStartSLOduration=2.248219325 podStartE2EDuration="2.248219325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:49:20.24750034 +0000 UTC m=+542.603896657" watchObservedRunningTime="2026-07-22 16:49:20.248219325 +0000 UTC m=+542.604615638" Jul 22 16:49:20.266056 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:20.266016 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" podStartSLOduration=2.266000284 podStartE2EDuration="2.266000284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:49:20.264958683 +0000 UTC m=+542.621355009" watchObservedRunningTime="2026-07-22 16:49:20.266000284 +0000 UTC m=+542.622396603" Jul 22 16:49:31.239214 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:31.239184 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-72667" Jul 22 16:49:31.239710 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:31.239261 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" Jul 22 16:49:34.083899 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.083859 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f"] Jul 22 16:49:34.084379 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.084158 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" podUID="095d417a-ad24-43b0-9415-cb35460eefce" containerName="manager" containerID="cri-o://16c66a98568849b628d679f7b53ea47703b4cb38131b6d0cdc2b856b951ab9be" gracePeriod=10 Jul 22 16:49:34.291831 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.291795 2577 generic.go:356] "Generic (PLEG): container finished" podID="095d417a-ad24-43b0-9415-cb35460eefce" containerID="16c66a98568849b628d679f7b53ea47703b4cb38131b6d0cdc2b856b951ab9be" exitCode=0 Jul 22 16:49:34.291970 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.291854 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" event={"ID":"095d417a-ad24-43b0-9415-cb35460eefce","Type":"ContainerDied","Data":"16c66a98568849b628d679f7b53ea47703b4cb38131b6d0cdc2b856b951ab9be"} Jul 22 16:49:34.334551 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.334493 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" Jul 22 16:49:34.352955 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.352921 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh"] Jul 22 16:49:34.353303 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.353288 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="095d417a-ad24-43b0-9415-cb35460eefce" containerName="manager" Jul 22 16:49:34.353357 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.353306 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="095d417a-ad24-43b0-9415-cb35460eefce" containerName="manager" Jul 22 16:49:34.353357 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.353355 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="095d417a-ad24-43b0-9415-cb35460eefce" containerName="manager" Jul 22 16:49:34.356335 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.356319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" Jul 22 16:49:34.373814 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.373781 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh"] Jul 22 16:49:34.472100 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.472066 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr8vr\" (UniqueName: \"kubernetes.io/projected/095d417a-ad24-43b0-9415-cb35460eefce-kube-api-access-hr8vr\") pod \"095d417a-ad24-43b0-9415-cb35460eefce\" (UID: \"095d417a-ad24-43b0-9415-cb35460eefce\") " Jul 22 16:49:34.472260 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.472118 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/095d417a-ad24-43b0-9415-cb35460eefce-extensions-socket-volume\") pod \"095d417a-ad24-43b0-9415-cb35460eefce\" (UID: \"095d417a-ad24-43b0-9415-cb35460eefce\") " Jul 22 16:49:34.472323 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.472299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6phj\" (UniqueName: \"kubernetes.io/projected/7b01f657-e976-45ef-867c-4999946a5b2a-kube-api-access-t6phj\") pod \"kuadrant-operator-controller-manager-86878c769f-hz6jh\" (UID: \"7b01f657-e976-45ef-867c-4999946a5b2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" Jul 22 16:49:34.472423 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.472403 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b01f657-e976-45ef-867c-4999946a5b2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hz6jh\" (UID: \"7b01f657-e976-45ef-867c-4999946a5b2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" Jul 22 16:49:34.472503 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.472473 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095d417a-ad24-43b0-9415-cb35460eefce-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "095d417a-ad24-43b0-9415-cb35460eefce" (UID: "095d417a-ad24-43b0-9415-cb35460eefce"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:49:34.474156 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.474134 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095d417a-ad24-43b0-9415-cb35460eefce-kube-api-access-hr8vr" (OuterVolumeSpecName: "kube-api-access-hr8vr") pod "095d417a-ad24-43b0-9415-cb35460eefce" (UID: "095d417a-ad24-43b0-9415-cb35460eefce"). InnerVolumeSpecName "kube-api-access-hr8vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:49:34.573848 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.573789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6phj\" (UniqueName: \"kubernetes.io/projected/7b01f657-e976-45ef-867c-4999946a5b2a-kube-api-access-t6phj\") pod \"kuadrant-operator-controller-manager-86878c769f-hz6jh\" (UID: \"7b01f657-e976-45ef-867c-4999946a5b2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" Jul 22 16:49:34.574042 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.573881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b01f657-e976-45ef-867c-4999946a5b2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hz6jh\" (UID: \"7b01f657-e976-45ef-867c-4999946a5b2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" Jul 22 16:49:34.574042 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.573956 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr8vr\" (UniqueName: \"kubernetes.io/projected/095d417a-ad24-43b0-9415-cb35460eefce-kube-api-access-hr8vr\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:49:34.574042 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.573971 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/095d417a-ad24-43b0-9415-cb35460eefce-extensions-socket-volume\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 16:49:34.574274 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.574253 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b01f657-e976-45ef-867c-4999946a5b2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hz6jh\" (UID: \"7b01f657-e976-45ef-867c-4999946a5b2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" Jul 22 16:49:34.582886 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.582862 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6phj\" (UniqueName: \"kubernetes.io/projected/7b01f657-e976-45ef-867c-4999946a5b2a-kube-api-access-t6phj\") pod \"kuadrant-operator-controller-manager-86878c769f-hz6jh\" (UID: \"7b01f657-e976-45ef-867c-4999946a5b2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" Jul 22 16:49:34.668846 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.668810 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" Jul 22 16:49:34.814061 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:34.813993 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh"] Jul 22 16:49:34.817072 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:49:34.817045 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b01f657_e976_45ef_867c_4999946a5b2a.slice/crio-4fff4683ee579fb48a826f1b8acb82334cbd130694e6b4c3a8fc1c58920af3b5 WatchSource:0}: Error finding container 4fff4683ee579fb48a826f1b8acb82334cbd130694e6b4c3a8fc1c58920af3b5: Status 404 returned error can't find the container with id 4fff4683ee579fb48a826f1b8acb82334cbd130694e6b4c3a8fc1c58920af3b5 Jul 22 16:49:35.297547 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:35.297516 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" Jul 22 16:49:35.298034 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:35.297547 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f" event={"ID":"095d417a-ad24-43b0-9415-cb35460eefce","Type":"ContainerDied","Data":"db23a993029874773fac03885c7d844720240f631087a956ffcc826e73d2cb6e"} Jul 22 16:49:35.298034 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:35.297625 2577 scope.go:117] "RemoveContainer" containerID="16c66a98568849b628d679f7b53ea47703b4cb38131b6d0cdc2b856b951ab9be" Jul 22 16:49:35.299114 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:35.299095 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" event={"ID":"7b01f657-e976-45ef-867c-4999946a5b2a","Type":"ContainerStarted","Data":"4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158"} Jul 22 16:49:35.299192 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:35.299133 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" event={"ID":"7b01f657-e976-45ef-867c-4999946a5b2a","Type":"ContainerStarted","Data":"4fff4683ee579fb48a826f1b8acb82334cbd130694e6b4c3a8fc1c58920af3b5"} Jul 22 16:49:35.299268 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:35.299253 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" Jul 22 16:49:35.328166 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:35.328108 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" podStartSLOduration=1.328091635 podStartE2EDuration="1.328091635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:49:35.326783127 +0000 UTC m=+557.683179457" watchObservedRunningTime="2026-07-22 16:49:35.328091635 +0000 UTC m=+557.684488021" Jul 22 16:49:35.340263 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:35.340228 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f"] Jul 22 16:49:35.344203 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:35.344174 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nx48f"] Jul 22 16:49:36.185528 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:36.185486 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="095d417a-ad24-43b0-9415-cb35460eefce" path="/var/lib/kubelet/pods/095d417a-ad24-43b0-9415-cb35460eefce/volumes" Jul 22 16:49:46.306805 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:46.306774 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" Jul 22 16:49:50.440420 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.440382 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw"] Jul 22 16:49:50.443159 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.443138 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.446085 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.446061 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pkf9x\"" Jul 22 16:49:50.455643 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.455618 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw"] Jul 22 16:49:50.612282 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.612243 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istio-token\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.612282 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.612287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.612543 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.612362 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-workload-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.612543 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.612402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-workload-certs\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.612543 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.612425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.612543 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.612486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.612543 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.612521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istio-data\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.612733 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.612553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-credential-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.612733 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.612602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjwqj\" (UniqueName: \"kubernetes.io/projected/0045fa9b-2030-4758-b2ff-3b95cc8245e7-kube-api-access-zjwqj\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.713933 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.713844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjwqj\" (UniqueName: \"kubernetes.io/projected/0045fa9b-2030-4758-b2ff-3b95cc8245e7-kube-api-access-zjwqj\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.713933 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.713891 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istio-token\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.713933 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.713923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.714154 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.713960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-workload-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.714154 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.713985 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-workload-certs\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.714154 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.714123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.714275 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.714217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.714275 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.714246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istio-data\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.714376 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.714276 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-credential-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.714424 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.714373 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-workload-certs\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.714500 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.714472 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-workload-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.714557 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.714513 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istio-data\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.714683 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.714663 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-credential-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.714806 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.714788 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.716476 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.716450 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.716599 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.716582 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.722817 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.722792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0045fa9b-2030-4758-b2ff-3b95cc8245e7-istio-token\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.723116 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.723089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjwqj\" (UniqueName: \"kubernetes.io/projected/0045fa9b-2030-4758-b2ff-3b95cc8245e7-kube-api-access-zjwqj\") pod \"maas-default-gateway-openshift-default-5df9b4f89-ghtbw\" (UID: \"0045fa9b-2030-4758-b2ff-3b95cc8245e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.754876 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.754843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:50.894170 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.894141 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw"] Jul 22 16:49:50.896903 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:49:50.896870 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0045fa9b_2030_4758_b2ff_3b95cc8245e7.slice/crio-a0cd84c906fb84214786d173e48de425721e3c236b945c3a33b6c2e09c3efacc WatchSource:0}: Error finding container a0cd84c906fb84214786d173e48de425721e3c236b945c3a33b6c2e09c3efacc: Status 404 returned error can't find the container with id a0cd84c906fb84214786d173e48de425721e3c236b945c3a33b6c2e09c3efacc Jul 22 16:49:50.899056 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.899021 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:49:50.899127 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.899098 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:49:50.899162 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:50.899129 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:49:51.368414 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:51.368375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" event={"ID":"0045fa9b-2030-4758-b2ff-3b95cc8245e7","Type":"ContainerStarted","Data":"aba2d1df0f9fa2c6ac57205e04a23764a4de79dff456b393ccdb5ae1eeb40687"} Jul 22 16:49:51.368615 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:51.368428 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" event={"ID":"0045fa9b-2030-4758-b2ff-3b95cc8245e7","Type":"ContainerStarted","Data":"a0cd84c906fb84214786d173e48de425721e3c236b945c3a33b6c2e09c3efacc"} Jul 22 16:49:51.391755 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:51.391683 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" podStartSLOduration=1.391665658 podStartE2EDuration="1.391665658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:49:51.390138133 +0000 UTC m=+573.746534453" watchObservedRunningTime="2026-07-22 16:49:51.391665658 +0000 UTC m=+573.748061976" Jul 22 16:49:51.756064 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:51.756027 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:51.760810 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:51.760785 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:52.373965 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:52.373936 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:49:52.374980 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:49:52.374960 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:50:06.660878 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.660843 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:50:06.664302 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.664278 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" Jul 22 16:50:06.666838 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.666814 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bfjkn\"" Jul 22 16:50:06.666838 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.666827 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 16:50:06.674399 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.674373 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:50:06.707363 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.707334 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:50:06.742306 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.742271 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtgb\" (UniqueName: \"kubernetes.io/projected/623d7e47-4b5b-40fb-a1cb-28e674413b68-kube-api-access-lbtgb\") pod \"limitador-limitador-54d6d988d5-nrqjk\" (UID: \"623d7e47-4b5b-40fb-a1cb-28e674413b68\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" Jul 22 16:50:06.742458 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.742322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/623d7e47-4b5b-40fb-a1cb-28e674413b68-config-file\") pod \"limitador-limitador-54d6d988d5-nrqjk\" (UID: \"623d7e47-4b5b-40fb-a1cb-28e674413b68\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" Jul 22 16:50:06.843167 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.843132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/623d7e47-4b5b-40fb-a1cb-28e674413b68-config-file\") pod \"limitador-limitador-54d6d988d5-nrqjk\" (UID: \"623d7e47-4b5b-40fb-a1cb-28e674413b68\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" Jul 22 16:50:06.843321 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.843297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtgb\" (UniqueName: \"kubernetes.io/projected/623d7e47-4b5b-40fb-a1cb-28e674413b68-kube-api-access-lbtgb\") pod \"limitador-limitador-54d6d988d5-nrqjk\" (UID: \"623d7e47-4b5b-40fb-a1cb-28e674413b68\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" Jul 22 16:50:06.843781 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.843764 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/623d7e47-4b5b-40fb-a1cb-28e674413b68-config-file\") pod \"limitador-limitador-54d6d988d5-nrqjk\" (UID: \"623d7e47-4b5b-40fb-a1cb-28e674413b68\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" Jul 22 16:50:06.851433 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.851410 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtgb\" (UniqueName: \"kubernetes.io/projected/623d7e47-4b5b-40fb-a1cb-28e674413b68-kube-api-access-lbtgb\") pod \"limitador-limitador-54d6d988d5-nrqjk\" (UID: \"623d7e47-4b5b-40fb-a1cb-28e674413b68\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" Jul 22 16:50:06.975378 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:06.975302 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" Jul 22 16:50:07.111826 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:07.111775 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:50:07.114246 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:50:07.114217 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623d7e47_4b5b_40fb_a1cb_28e674413b68.slice/crio-b6a6f1a7a2cde32816b3b961e9af7120ff9073ee11898ce9a8c13285f8f9e3da WatchSource:0}: Error finding container b6a6f1a7a2cde32816b3b961e9af7120ff9073ee11898ce9a8c13285f8f9e3da: Status 404 returned error can't find the container with id b6a6f1a7a2cde32816b3b961e9af7120ff9073ee11898ce9a8c13285f8f9e3da Jul 22 16:50:07.441540 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:07.441504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" event={"ID":"623d7e47-4b5b-40fb-a1cb-28e674413b68","Type":"ContainerStarted","Data":"b6a6f1a7a2cde32816b3b961e9af7120ff9073ee11898ce9a8c13285f8f9e3da"} Jul 22 16:50:10.456680 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:10.456646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" event={"ID":"623d7e47-4b5b-40fb-a1cb-28e674413b68","Type":"ContainerStarted","Data":"cdc1d4c705275ac547b7b3d2904e2267f93b2c589d020ee21be4c9011a59c893"} Jul 22 16:50:10.457123 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:10.456757 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" Jul 22 16:50:10.474118 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:10.474075 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" podStartSLOduration=2.017140843 podStartE2EDuration="4.474061206s" totalImagesPullingTime="2.456920363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:50:10.473237269 +0000 UTC m=+592.829633588" watchObservedRunningTime="2026-07-22 16:50:10.474061206 +0000 UTC m=+592.830457524" Jul 22 16:50:21.463294 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:50:21.463261 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nrqjk" Jul 22 16:51:20.511788 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:20.511705 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:51:27.784463 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:27.784424 2577 generic.go:356] "Generic (PLEG): container finished" podID="0045fa9b-2030-4758-b2ff-3b95cc8245e7" containerID="aba2d1df0f9fa2c6ac57205e04a23764a4de79dff456b393ccdb5ae1eeb40687" exitCode=0 Jul 22 16:51:27.784967 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:27.784472 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" event={"ID":"0045fa9b-2030-4758-b2ff-3b95cc8245e7","Type":"ContainerDied","Data":"aba2d1df0f9fa2c6ac57205e04a23764a4de79dff456b393ccdb5ae1eeb40687"} Jul 22 16:51:27.785034 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:27.784970 2577 scope.go:117] "RemoveContainer" containerID="aba2d1df0f9fa2c6ac57205e04a23764a4de79dff456b393ccdb5ae1eeb40687" Jul 22 16:51:27.785949 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:27.785922 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:51:27.786042 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:27.785992 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:51:27.786089 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:27.786051 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 16:51:28.790478 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:28.790443 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" event={"ID":"0045fa9b-2030-4758-b2ff-3b95cc8245e7","Type":"ContainerStarted","Data":"369c8c2549273bf0627191ed0ae88e4342cfeb0a930135f85adeda9aea8920ce"} Jul 22 16:51:29.755956 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:29.755914 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:51:29.756285 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:29.756247 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-ghtbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 22 16:51:29.756386 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:29.756322 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" podUID="0045fa9b-2030-4758-b2ff-3b95cc8245e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 22 16:51:30.756014 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:30.755974 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-ghtbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 22 16:51:30.756975 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:30.756557 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" podUID="0045fa9b-2030-4758-b2ff-3b95cc8245e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 22 16:51:31.755803 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:31.755766 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-ghtbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 22 16:51:31.756242 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:31.756209 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" podUID="0045fa9b-2030-4758-b2ff-3b95cc8245e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 22 16:51:32.759365 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:32.759330 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:51:32.759805 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:32.759631 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:51:32.760344 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:32.760328 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-ghtbw" Jul 22 16:51:57.991908 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:51:57.991875 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:52:13.095970 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:13.095876 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:52:21.187955 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:21.187920 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:52:25.619033 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:25.619001 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:52:32.514018 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.513987 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg"] Jul 22 16:52:32.516919 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.516902 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.519278 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.519255 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 16:52:32.520156 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.520131 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 22 16:52:32.520156 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.520131 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7d4dj\"" Jul 22 16:52:32.520304 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.520141 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 16:52:32.527934 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.527914 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg"] Jul 22 16:52:32.667149 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.667119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66d566b-e152-46eb-8ea9-7e14c003af16-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.667149 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.667153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chnsb\" (UniqueName: \"kubernetes.io/projected/e66d566b-e152-46eb-8ea9-7e14c003af16-kube-api-access-chnsb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.667389 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.667175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66d566b-e152-46eb-8ea9-7e14c003af16-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.667389 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.667228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66d566b-e152-46eb-8ea9-7e14c003af16-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.667471 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.667411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66d566b-e152-46eb-8ea9-7e14c003af16-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.667505 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.667466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66d566b-e152-46eb-8ea9-7e14c003af16-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.768491 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.768423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66d566b-e152-46eb-8ea9-7e14c003af16-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.768491 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.768464 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66d566b-e152-46eb-8ea9-7e14c003af16-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.768659 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.768579 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66d566b-e152-46eb-8ea9-7e14c003af16-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.768659 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.768644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66d566b-e152-46eb-8ea9-7e14c003af16-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.768764 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.768666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chnsb\" (UniqueName: \"kubernetes.io/projected/e66d566b-e152-46eb-8ea9-7e14c003af16-kube-api-access-chnsb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.768764 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.768684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66d566b-e152-46eb-8ea9-7e14c003af16-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.768875 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.768810 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66d566b-e152-46eb-8ea9-7e14c003af16-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.768875 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.768844 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66d566b-e152-46eb-8ea9-7e14c003af16-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.769025 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.769005 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66d566b-e152-46eb-8ea9-7e14c003af16-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.770758 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.770726 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66d566b-e152-46eb-8ea9-7e14c003af16-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.771136 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.771118 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66d566b-e152-46eb-8ea9-7e14c003af16-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.777476 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.777447 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chnsb\" (UniqueName: \"kubernetes.io/projected/e66d566b-e152-46eb-8ea9-7e14c003af16-kube-api-access-chnsb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gzzqg\" (UID: \"e66d566b-e152-46eb-8ea9-7e14c003af16\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.827752 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.827709 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:32.956830 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.956804 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg"] Jul 22 16:52:32.959259 ip-10-0-136-75 kubenswrapper[2577]: W0722 16:52:32.959231 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66d566b_e152_46eb_8ea9_7e14c003af16.slice/crio-d7957c99a1ea4af7b4dcb8cdf2ab2c44ff64a6d1fff428bdeb915204ce71ade6 WatchSource:0}: Error finding container d7957c99a1ea4af7b4dcb8cdf2ab2c44ff64a6d1fff428bdeb915204ce71ade6: Status 404 returned error can't find the container with id d7957c99a1ea4af7b4dcb8cdf2ab2c44ff64a6d1fff428bdeb915204ce71ade6 Jul 22 16:52:32.960587 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:32.960565 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:52:33.072303 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:33.072230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" event={"ID":"e66d566b-e152-46eb-8ea9-7e14c003af16","Type":"ContainerStarted","Data":"d7957c99a1ea4af7b4dcb8cdf2ab2c44ff64a6d1fff428bdeb915204ce71ade6"} Jul 22 16:52:34.077886 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:34.077855 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" event={"ID":"e66d566b-e152-46eb-8ea9-7e14c003af16","Type":"ContainerStarted","Data":"ed5805759eab7d2f2e8e2a2c54f4c8cc359d079f0102cd2143ff0e590e6e36cb"} Jul 22 16:52:34.096941 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:34.096897 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" podStartSLOduration=1.051495586 podStartE2EDuration="2.096881423s" totalImagesPullingTime="1.045385837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:52:34.09505554 +0000 UTC m=+736.451451905" watchObservedRunningTime="2026-07-22 16:52:34.096881423 +0000 UTC m=+736.453277774" Jul 22 16:52:35.308168 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:35.308126 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:52:42.828344 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:42.828292 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:42.828820 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:42.828371 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:42.843446 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:42.843404 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:43.134616 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:43.134525 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gzzqg" Jul 22 16:52:48.428385 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:52:48.428346 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:54:11.101036 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:54:11.100995 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:54:21.296084 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:54:21.296051 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:54:31.996491 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:54:31.996455 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:56:12.796588 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:56:12.796550 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:56:24.398035 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:56:24.398000 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:56:59.692554 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:56:59.692476 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:57:08.893681 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:57:08.893649 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:57:19.101951 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:57:19.101911 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:57:26.791342 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:57:26.791305 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:57:36.201276 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:57:36.201245 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:57:52.203725 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:57:52.203692 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:57:58.404189 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:57:58.404154 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:58:07.795314 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:58:07.795274 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:58:29.305905 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:58:29.305823 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:58:40.001520 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:58:40.001481 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:59:03.200662 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:59:03.200624 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:59:13.093492 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:59:13.093460 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:59:28.807577 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:59:28.807541 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:59:38.806127 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:59:38.806093 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 16:59:51.601196 ip-10-0-136-75 kubenswrapper[2577]: I0722 16:59:51.601123 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:00:35.788165 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:35.788134 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh"] Jul 22 17:00:35.788647 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:35.788354 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" podUID="7b01f657-e976-45ef-867c-4999946a5b2a" containerName="manager" containerID="cri-o://4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158" gracePeriod=10 Jul 22 17:00:39.037117 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.037091 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" Jul 22 17:00:39.101938 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.101863 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b01f657-e976-45ef-867c-4999946a5b2a-extensions-socket-volume\") pod \"7b01f657-e976-45ef-867c-4999946a5b2a\" (UID: \"7b01f657-e976-45ef-867c-4999946a5b2a\") " Jul 22 17:00:39.101938 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.101912 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6phj\" (UniqueName: \"kubernetes.io/projected/7b01f657-e976-45ef-867c-4999946a5b2a-kube-api-access-t6phj\") pod \"7b01f657-e976-45ef-867c-4999946a5b2a\" (UID: \"7b01f657-e976-45ef-867c-4999946a5b2a\") " Jul 22 17:00:39.102260 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.102234 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b01f657-e976-45ef-867c-4999946a5b2a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7b01f657-e976-45ef-867c-4999946a5b2a" (UID: "7b01f657-e976-45ef-867c-4999946a5b2a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:00:39.103877 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.103852 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b01f657-e976-45ef-867c-4999946a5b2a-kube-api-access-t6phj" (OuterVolumeSpecName: "kube-api-access-t6phj") pod "7b01f657-e976-45ef-867c-4999946a5b2a" (UID: "7b01f657-e976-45ef-867c-4999946a5b2a"). InnerVolumeSpecName "kube-api-access-t6phj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:00:39.202629 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.202603 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b01f657-e976-45ef-867c-4999946a5b2a-extensions-socket-volume\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:00:39.202629 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.202626 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6phj\" (UniqueName: \"kubernetes.io/projected/7b01f657-e976-45ef-867c-4999946a5b2a-kube-api-access-t6phj\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:00:39.212402 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.212377 2577 generic.go:356] "Generic (PLEG): container finished" podID="7b01f657-e976-45ef-867c-4999946a5b2a" containerID="4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158" exitCode=0 Jul 22 17:00:39.212518 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.212419 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" Jul 22 17:00:39.212518 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.212444 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" event={"ID":"7b01f657-e976-45ef-867c-4999946a5b2a","Type":"ContainerDied","Data":"4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158"} Jul 22 17:00:39.212518 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.212495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh" event={"ID":"7b01f657-e976-45ef-867c-4999946a5b2a","Type":"ContainerDied","Data":"4fff4683ee579fb48a826f1b8acb82334cbd130694e6b4c3a8fc1c58920af3b5"} Jul 22 17:00:39.212671 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.212525 2577 scope.go:117] "RemoveContainer" containerID="4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158" Jul 22 17:00:39.222471 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.222444 2577 scope.go:117] "RemoveContainer" containerID="4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158" Jul 22 17:00:39.222720 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:00:39.222701 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158\": container with ID starting with 4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158 not found: ID does not exist" containerID="4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158" Jul 22 17:00:39.222811 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.222747 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158"} err="failed to get container status \"4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158\": rpc error: code = NotFound desc = could not find container \"4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158\": container with ID starting with 4b89b466e1c1c42743c82c8839a22b27ce178cdcec932d4e0a4a4cf54a724158 not found: ID does not exist" Jul 22 17:00:39.235020 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.235000 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh"] Jul 22 17:00:39.240603 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:39.240582 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hz6jh"] Jul 22 17:00:40.185201 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:00:40.185170 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b01f657-e976-45ef-867c-4999946a5b2a" path="/var/lib/kubelet/pods/7b01f657-e976-45ef-867c-4999946a5b2a/volumes" Jul 22 17:01:37.080146 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.080111 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s"] Jul 22 17:01:37.080577 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.080433 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b01f657-e976-45ef-867c-4999946a5b2a" containerName="manager" Jul 22 17:01:37.080577 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.080443 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b01f657-e976-45ef-867c-4999946a5b2a" containerName="manager" Jul 22 17:01:37.080577 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.080499 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b01f657-e976-45ef-867c-4999946a5b2a" containerName="manager" Jul 22 17:01:37.084954 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.084935 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" Jul 22 17:01:37.088087 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.088064 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ctrlr\"" Jul 22 17:01:37.104673 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.104648 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s"] Jul 22 17:01:37.147946 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.147906 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnms\" (UniqueName: \"kubernetes.io/projected/4994b74d-442b-4da1-a066-42e5008afd9f-kube-api-access-rfnms\") pod \"kuadrant-operator-controller-manager-86878c769f-dc94s\" (UID: \"4994b74d-442b-4da1-a066-42e5008afd9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" Jul 22 17:01:37.148092 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.148004 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4994b74d-442b-4da1-a066-42e5008afd9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dc94s\" (UID: \"4994b74d-442b-4da1-a066-42e5008afd9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" Jul 22 17:01:37.249350 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.249314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4994b74d-442b-4da1-a066-42e5008afd9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dc94s\" (UID: \"4994b74d-442b-4da1-a066-42e5008afd9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" Jul 22 17:01:37.249507 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.249376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnms\" (UniqueName: \"kubernetes.io/projected/4994b74d-442b-4da1-a066-42e5008afd9f-kube-api-access-rfnms\") pod \"kuadrant-operator-controller-manager-86878c769f-dc94s\" (UID: \"4994b74d-442b-4da1-a066-42e5008afd9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" Jul 22 17:01:37.249696 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.249678 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4994b74d-442b-4da1-a066-42e5008afd9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dc94s\" (UID: \"4994b74d-442b-4da1-a066-42e5008afd9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" Jul 22 17:01:37.258243 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.258216 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnms\" (UniqueName: \"kubernetes.io/projected/4994b74d-442b-4da1-a066-42e5008afd9f-kube-api-access-rfnms\") pod \"kuadrant-operator-controller-manager-86878c769f-dc94s\" (UID: \"4994b74d-442b-4da1-a066-42e5008afd9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" Jul 22 17:01:37.394359 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.394279 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" Jul 22 17:01:37.528622 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:37.528593 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s"] Jul 22 17:01:37.532424 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:01:37.532397 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4994b74d_442b_4da1_a066_42e5008afd9f.slice/crio-66b35e00f6f86a58d209346706e0e42171c7abc9e5154150184907ccb28a8bf8 WatchSource:0}: Error finding container 66b35e00f6f86a58d209346706e0e42171c7abc9e5154150184907ccb28a8bf8: Status 404 returned error can't find the container with id 66b35e00f6f86a58d209346706e0e42171c7abc9e5154150184907ccb28a8bf8 Jul 22 17:01:38.466231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:38.466196 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" event={"ID":"4994b74d-442b-4da1-a066-42e5008afd9f","Type":"ContainerStarted","Data":"b17364e034dd60c4f0e085e88d28ac1f9ecc3b5054e6a308e5cb501bc949013f"} Jul 22 17:01:38.466619 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:38.466248 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" event={"ID":"4994b74d-442b-4da1-a066-42e5008afd9f","Type":"ContainerStarted","Data":"66b35e00f6f86a58d209346706e0e42171c7abc9e5154150184907ccb28a8bf8"} Jul 22 17:01:38.466619 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:38.466277 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" Jul 22 17:01:38.489479 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:38.489434 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" podStartSLOduration=1.489419356 podStartE2EDuration="1.489419356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:01:38.487128637 +0000 UTC m=+1280.843524976" watchObservedRunningTime="2026-07-22 17:01:38.489419356 +0000 UTC m=+1280.845815673" Jul 22 17:01:49.473331 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:49.473300 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dc94s" Jul 22 17:01:57.192050 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:01:57.192011 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:02:06.089958 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:02:06.089919 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:02:18.407997 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:02:18.407970 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:02:29.799394 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:02:29.799360 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:02:41.222108 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:02:41.222031 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:02:51.139265 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:02:51.139226 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:03:01.505991 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:03:01.505958 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:03:10.926143 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:03:10.926112 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:03:22.525098 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:03:22.525063 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:03:31.094598 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:03:31.094561 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:03:43.098522 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:03:43.098485 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:04:04.114316 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:04.114275 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:04:13.376058 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:13.375976 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:04:16.401869 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.401835 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj"] Jul 22 17:04:16.405657 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.405636 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.408044 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.408021 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c6ba3d72-kserve-self-signed-certs\"" Jul 22 17:04:16.423092 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.423073 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj"] Jul 22 17:04:16.449051 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.449029 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-tmp-dir\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.449150 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.449073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-home\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.449150 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.449130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54dca86-4d83-4642-a0c2-35a0a4562069-tls-certs\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.449231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.449155 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjm4\" (UniqueName: \"kubernetes.io/projected/a54dca86-4d83-4642-a0c2-35a0a4562069-kube-api-access-7bjm4\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.449231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.449182 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-dshm\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.449231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.449223 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-model-cache\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.550587 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.550553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-home\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.550707 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.550606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54dca86-4d83-4642-a0c2-35a0a4562069-tls-certs\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.550707 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.550637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjm4\" (UniqueName: \"kubernetes.io/projected/a54dca86-4d83-4642-a0c2-35a0a4562069-kube-api-access-7bjm4\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.550707 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.550656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-dshm\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.550707 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.550702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-model-cache\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.550992 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.550765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-tmp-dir\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.550992 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.550935 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-home\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.551109 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.551047 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-model-cache\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.551149 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.551112 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-tmp-dir\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.552894 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.552870 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-dshm\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.553214 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.553198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54dca86-4d83-4642-a0c2-35a0a4562069-tls-certs\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.557958 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.557939 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjm4\" (UniqueName: \"kubernetes.io/projected/a54dca86-4d83-4642-a0c2-35a0a4562069-kube-api-access-7bjm4\") pod \"e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.714560 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.714487 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:16.843878 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:16.843853 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj"] Jul 22 17:04:16.846424 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:04:16.846399 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda54dca86_4d83_4642_a0c2_35a0a4562069.slice/crio-9060badad22c054f8a74cde352e7c05dd5c217f7c777a74f6a4df09f6b1d29d3 WatchSource:0}: Error finding container 9060badad22c054f8a74cde352e7c05dd5c217f7c777a74f6a4df09f6b1d29d3: Status 404 returned error can't find the container with id 9060badad22c054f8a74cde352e7c05dd5c217f7c777a74f6a4df09f6b1d29d3 Jul 22 17:04:17.166333 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:17.166300 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" event={"ID":"a54dca86-4d83-4642-a0c2-35a0a4562069","Type":"ContainerStarted","Data":"6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b"} Jul 22 17:04:17.166333 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:17.166339 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" event={"ID":"a54dca86-4d83-4642-a0c2-35a0a4562069","Type":"ContainerStarted","Data":"9060badad22c054f8a74cde352e7c05dd5c217f7c777a74f6a4df09f6b1d29d3"} Jul 22 17:04:17.186789 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:17.186719 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" podStartSLOduration=1.186703221 podStartE2EDuration="1.186703221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:04:17.185368038 +0000 UTC m=+1439.541764356" watchObservedRunningTime="2026-07-22 17:04:17.186703221 +0000 UTC m=+1439.543099538" Jul 22 17:04:22.681427 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.681396 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5"] Jul 22 17:04:22.683723 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.683702 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.686761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.686716 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c6ba3d72-kserve-self-signed-certs\"" Jul 22 17:04:22.697076 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.697052 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5"] Jul 22 17:04:22.803425 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.803401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-tmp-dir\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.803559 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.803445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-model-cache\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.803559 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.803470 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-dshm\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.803559 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.803537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2486e42e-7702-4ea1-b053-4f5db39fe592-tls-certs\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.803658 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.803588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-959l4\" (UniqueName: \"kubernetes.io/projected/2486e42e-7702-4ea1-b053-4f5db39fe592-kube-api-access-959l4\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.803658 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.803637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-home\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.904502 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.904474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2486e42e-7702-4ea1-b053-4f5db39fe592-tls-certs\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.904636 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.904524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-959l4\" (UniqueName: \"kubernetes.io/projected/2486e42e-7702-4ea1-b053-4f5db39fe592-kube-api-access-959l4\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.904636 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.904544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-home\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.904636 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.904590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-tmp-dir\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.904636 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.904619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-model-cache\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.904843 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.904639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-dshm\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.905076 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.905058 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-tmp-dir\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.905176 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.905149 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-home\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.905231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.905203 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-model-cache\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.906772 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.906729 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-dshm\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.907188 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.907167 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2486e42e-7702-4ea1-b053-4f5db39fe592-tls-certs\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.916107 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.916082 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-959l4\" (UniqueName: \"kubernetes.io/projected/2486e42e-7702-4ea1-b053-4f5db39fe592-kube-api-access-959l4\") pod \"e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:22.993243 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:22.993172 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:23.131385 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:23.131357 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5"] Jul 22 17:04:23.133109 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:04:23.133084 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2486e42e_7702_4ea1_b053_4f5db39fe592.slice/crio-4f355d126986536384cb4b251a19f8af0c1bddb2a15a7eccd672105065fa67b9 WatchSource:0}: Error finding container 4f355d126986536384cb4b251a19f8af0c1bddb2a15a7eccd672105065fa67b9: Status 404 returned error can't find the container with id 4f355d126986536384cb4b251a19f8af0c1bddb2a15a7eccd672105065fa67b9 Jul 22 17:04:23.195778 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:23.195730 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" event={"ID":"2486e42e-7702-4ea1-b053-4f5db39fe592","Type":"ContainerStarted","Data":"a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312"} Jul 22 17:04:23.195919 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:23.195799 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" event={"ID":"2486e42e-7702-4ea1-b053-4f5db39fe592","Type":"ContainerStarted","Data":"4f355d126986536384cb4b251a19f8af0c1bddb2a15a7eccd672105065fa67b9"} Jul 22 17:04:23.220526 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:23.220476 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" podStartSLOduration=1.220461015 podStartE2EDuration="1.220461015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:04:23.218419009 +0000 UTC m=+1445.574815318" watchObservedRunningTime="2026-07-22 17:04:23.220461015 +0000 UTC m=+1445.576857332" Jul 22 17:04:26.714964 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:26.714926 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:26.714964 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:26.714974 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:26.727673 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:26.727649 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:27.224648 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:27.224620 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:04:30.304132 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:30.304096 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:04:32.994119 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:32.994083 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:32.994119 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:32.994122 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:33.006567 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:33.006541 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:33.250867 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:33.250789 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:04:48.404062 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:48.404032 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj"] Jul 22 17:04:49.883174 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:49.883145 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5"] Jul 22 17:04:52.488141 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:04:52.488107 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:05:03.423579 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.423536 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" podUID="a54dca86-4d83-4642-a0c2-35a0a4562069" containerName="main" containerID="cri-o://6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b" gracePeriod=45 Jul 22 17:05:03.678559 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.678503 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:05:03.753030 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.752996 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-home\") pod \"a54dca86-4d83-4642-a0c2-35a0a4562069\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " Jul 22 17:05:03.753200 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.753040 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bjm4\" (UniqueName: \"kubernetes.io/projected/a54dca86-4d83-4642-a0c2-35a0a4562069-kube-api-access-7bjm4\") pod \"a54dca86-4d83-4642-a0c2-35a0a4562069\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " Jul 22 17:05:03.753200 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.753089 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-tmp-dir\") pod \"a54dca86-4d83-4642-a0c2-35a0a4562069\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " Jul 22 17:05:03.753200 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.753135 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-model-cache\") pod \"a54dca86-4d83-4642-a0c2-35a0a4562069\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " Jul 22 17:05:03.753200 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.753175 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54dca86-4d83-4642-a0c2-35a0a4562069-tls-certs\") pod \"a54dca86-4d83-4642-a0c2-35a0a4562069\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " Jul 22 17:05:03.753200 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.753190 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-dshm\") pod \"a54dca86-4d83-4642-a0c2-35a0a4562069\" (UID: \"a54dca86-4d83-4642-a0c2-35a0a4562069\") " Jul 22 17:05:03.753450 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.753249 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-home" (OuterVolumeSpecName: "home") pod "a54dca86-4d83-4642-a0c2-35a0a4562069" (UID: "a54dca86-4d83-4642-a0c2-35a0a4562069"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:05:03.753450 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.753353 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a54dca86-4d83-4642-a0c2-35a0a4562069" (UID: "a54dca86-4d83-4642-a0c2-35a0a4562069"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:05:03.753450 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.753427 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-model-cache" (OuterVolumeSpecName: "model-cache") pod "a54dca86-4d83-4642-a0c2-35a0a4562069" (UID: "a54dca86-4d83-4642-a0c2-35a0a4562069"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:05:03.753450 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.753442 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-home\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:05:03.753581 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.753454 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-tmp-dir\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:05:03.755346 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.755314 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-dshm" (OuterVolumeSpecName: "dshm") pod "a54dca86-4d83-4642-a0c2-35a0a4562069" (UID: "a54dca86-4d83-4642-a0c2-35a0a4562069"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:05:03.755448 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.755375 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54dca86-4d83-4642-a0c2-35a0a4562069-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a54dca86-4d83-4642-a0c2-35a0a4562069" (UID: "a54dca86-4d83-4642-a0c2-35a0a4562069"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:05:03.755448 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.755405 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54dca86-4d83-4642-a0c2-35a0a4562069-kube-api-access-7bjm4" (OuterVolumeSpecName: "kube-api-access-7bjm4") pod "a54dca86-4d83-4642-a0c2-35a0a4562069" (UID: "a54dca86-4d83-4642-a0c2-35a0a4562069"). InnerVolumeSpecName "kube-api-access-7bjm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:05:03.854550 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.854512 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54dca86-4d83-4642-a0c2-35a0a4562069-tls-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:05:03.854550 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.854542 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-dshm\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:05:03.854550 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.854555 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bjm4\" (UniqueName: \"kubernetes.io/projected/a54dca86-4d83-4642-a0c2-35a0a4562069-kube-api-access-7bjm4\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:05:03.854818 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:03.854567 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54dca86-4d83-4642-a0c2-35a0a4562069-model-cache\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:05:04.382329 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:04.382292 2577 generic.go:356] "Generic (PLEG): container finished" podID="a54dca86-4d83-4642-a0c2-35a0a4562069" containerID="6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b" exitCode=0 Jul 22 17:05:04.382492 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:04.382337 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" Jul 22 17:05:04.382492 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:04.382378 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" event={"ID":"a54dca86-4d83-4642-a0c2-35a0a4562069","Type":"ContainerDied","Data":"6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b"} Jul 22 17:05:04.382492 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:04.382424 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj" event={"ID":"a54dca86-4d83-4642-a0c2-35a0a4562069","Type":"ContainerDied","Data":"9060badad22c054f8a74cde352e7c05dd5c217f7c777a74f6a4df09f6b1d29d3"} Jul 22 17:05:04.382492 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:04.382441 2577 scope.go:117] "RemoveContainer" containerID="6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b" Jul 22 17:05:04.390596 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:04.390582 2577 scope.go:117] "RemoveContainer" containerID="6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b" Jul 22 17:05:04.390951 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:05:04.390932 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b\": container with ID starting with 6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b not found: ID does not exist" containerID="6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b" Jul 22 17:05:04.391019 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:04.390959 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b"} err="failed to get container status \"6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b\": rpc error: code = NotFound desc = could not find container \"6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b\": container with ID starting with 6ec7ad72b918ded5ee0215c479a918e3a2447ef8a6ea7fa8a9a9cda96737b36b not found: ID does not exist" Jul 22 17:05:04.409674 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:04.406480 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj"] Jul 22 17:05:04.416886 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:04.416864 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c6ba3d72-kserve-7899dcf759-wfqdj"] Jul 22 17:05:04.903283 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:04.903223 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" podUID="2486e42e-7702-4ea1-b053-4f5db39fe592" containerName="main" containerID="cri-o://a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312" gracePeriod=45 Jul 22 17:05:05.148591 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.148567 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:05:05.164629 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.164577 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-959l4\" (UniqueName: \"kubernetes.io/projected/2486e42e-7702-4ea1-b053-4f5db39fe592-kube-api-access-959l4\") pod \"2486e42e-7702-4ea1-b053-4f5db39fe592\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " Jul 22 17:05:05.164629 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.164610 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-model-cache\") pod \"2486e42e-7702-4ea1-b053-4f5db39fe592\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " Jul 22 17:05:05.164784 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.164636 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-dshm\") pod \"2486e42e-7702-4ea1-b053-4f5db39fe592\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " Jul 22 17:05:05.164784 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.164664 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-tmp-dir\") pod \"2486e42e-7702-4ea1-b053-4f5db39fe592\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " Jul 22 17:05:05.164784 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.164708 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2486e42e-7702-4ea1-b053-4f5db39fe592-tls-certs\") pod \"2486e42e-7702-4ea1-b053-4f5db39fe592\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " Jul 22 17:05:05.164784 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.164757 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-home\") pod \"2486e42e-7702-4ea1-b053-4f5db39fe592\" (UID: \"2486e42e-7702-4ea1-b053-4f5db39fe592\") " Jul 22 17:05:05.164989 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.164920 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-model-cache" (OuterVolumeSpecName: "model-cache") pod "2486e42e-7702-4ea1-b053-4f5db39fe592" (UID: "2486e42e-7702-4ea1-b053-4f5db39fe592"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:05:05.165051 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.164987 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2486e42e-7702-4ea1-b053-4f5db39fe592" (UID: "2486e42e-7702-4ea1-b053-4f5db39fe592"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:05:05.165051 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.165003 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-home" (OuterVolumeSpecName: "home") pod "2486e42e-7702-4ea1-b053-4f5db39fe592" (UID: "2486e42e-7702-4ea1-b053-4f5db39fe592"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:05:05.165151 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.165074 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-tmp-dir\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:05:05.165151 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.165087 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-model-cache\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:05:05.166746 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.166713 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-dshm" (OuterVolumeSpecName: "dshm") pod "2486e42e-7702-4ea1-b053-4f5db39fe592" (UID: "2486e42e-7702-4ea1-b053-4f5db39fe592"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:05:05.166866 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.166849 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2486e42e-7702-4ea1-b053-4f5db39fe592-kube-api-access-959l4" (OuterVolumeSpecName: "kube-api-access-959l4") pod "2486e42e-7702-4ea1-b053-4f5db39fe592" (UID: "2486e42e-7702-4ea1-b053-4f5db39fe592"). InnerVolumeSpecName "kube-api-access-959l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:05:05.166907 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.166875 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2486e42e-7702-4ea1-b053-4f5db39fe592-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2486e42e-7702-4ea1-b053-4f5db39fe592" (UID: "2486e42e-7702-4ea1-b053-4f5db39fe592"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:05:05.266362 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.266338 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-959l4\" (UniqueName: \"kubernetes.io/projected/2486e42e-7702-4ea1-b053-4f5db39fe592-kube-api-access-959l4\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:05:05.266362 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.266360 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-dshm\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:05:05.266500 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.266370 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2486e42e-7702-4ea1-b053-4f5db39fe592-tls-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:05:05.266500 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.266379 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2486e42e-7702-4ea1-b053-4f5db39fe592-home\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:05:05.387910 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.387872 2577 generic.go:356] "Generic (PLEG): container finished" podID="2486e42e-7702-4ea1-b053-4f5db39fe592" containerID="a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312" exitCode=0 Jul 22 17:05:05.388022 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.387927 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" Jul 22 17:05:05.388022 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.387956 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" event={"ID":"2486e42e-7702-4ea1-b053-4f5db39fe592","Type":"ContainerDied","Data":"a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312"} Jul 22 17:05:05.388022 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.387995 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5" event={"ID":"2486e42e-7702-4ea1-b053-4f5db39fe592","Type":"ContainerDied","Data":"4f355d126986536384cb4b251a19f8af0c1bddb2a15a7eccd672105065fa67b9"} Jul 22 17:05:05.388022 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.388012 2577 scope.go:117] "RemoveContainer" containerID="a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312" Jul 22 17:05:05.397180 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.397163 2577 scope.go:117] "RemoveContainer" containerID="a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312" Jul 22 17:05:05.397464 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:05:05.397448 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312\": container with ID starting with a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312 not found: ID does not exist" containerID="a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312" Jul 22 17:05:05.397501 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.397475 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312"} err="failed to get container status \"a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312\": rpc error: code = NotFound desc = could not find container \"a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312\": container with ID starting with a664e47f9dc3b07ba4bb9f55201c7a8d8c99cf521d880be4ca422b58471a6312 not found: ID does not exist" Jul 22 17:05:05.409164 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.409138 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5"] Jul 22 17:05:05.415433 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.415382 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c6ba3d72-kserve-fc648665f-792h5"] Jul 22 17:05:05.610752 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:05.610707 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:05:06.185630 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:06.185596 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2486e42e-7702-4ea1-b053-4f5db39fe592" path="/var/lib/kubelet/pods/2486e42e-7702-4ea1-b053-4f5db39fe592/volumes" Jul 22 17:05:06.186071 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:06.186019 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a54dca86-4d83-4642-a0c2-35a0a4562069" path="/var/lib/kubelet/pods/a54dca86-4d83-4642-a0c2-35a0a4562069/volumes" Jul 22 17:05:15.608295 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:15.608262 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:05:27.394483 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:27.394449 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:05:38.610165 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:38.610132 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:05:47.795961 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:47.795875 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:05:58.194919 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:05:58.194886 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:06:07.495128 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:06:07.495093 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:06:16.493943 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:06:16.493908 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:06:29.602408 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:06:29.602372 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:06:38.293381 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:06:38.293348 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:06:47.701809 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:06:47.701777 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:06:58.997066 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:06:58.997029 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:07:11.500872 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:07:11.500786 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:07:20.297159 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:07:20.297123 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:07:32.104220 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:07:32.104184 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:07:43.834005 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:07:43.833967 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:07:52.603892 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:07:52.603858 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:08:00.907753 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:00.907698 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:08:11.902880 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:11.902842 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:08:34.274916 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:34.274877 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:08:40.022492 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.022453 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg"] Jul 22 17:08:40.023136 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.022833 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2486e42e-7702-4ea1-b053-4f5db39fe592" containerName="main" Jul 22 17:08:40.023136 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.022850 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2486e42e-7702-4ea1-b053-4f5db39fe592" containerName="main" Jul 22 17:08:40.023136 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.022882 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54dca86-4d83-4642-a0c2-35a0a4562069" containerName="main" Jul 22 17:08:40.023136 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.022888 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54dca86-4d83-4642-a0c2-35a0a4562069" containerName="main" Jul 22 17:08:40.023136 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.022939 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a54dca86-4d83-4642-a0c2-35a0a4562069" containerName="main" Jul 22 17:08:40.023136 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.022946 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2486e42e-7702-4ea1-b053-4f5db39fe592" containerName="main" Jul 22 17:08:40.025717 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.025695 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.029707 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.029688 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-76a9cf43-openshift-default-dockercfg-rsfbd\"" Jul 22 17:08:40.055682 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.055643 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg"] Jul 22 17:08:40.123984 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.123952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-data\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.124148 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.124005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-workload-certs\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.124148 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.124035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-podinfo\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.124148 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.124084 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-workload-socket\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.124148 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.124126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-envoy\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.124286 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.124193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmjrd\" (UniqueName: \"kubernetes.io/projected/6f238afe-1ad4-4fca-a941-ebd07d08cf87-kube-api-access-cmjrd\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.124286 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.124221 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-token\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.124286 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.124238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istiod-ca-cert\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.124286 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.124253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-credential-socket\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225134 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmjrd\" (UniqueName: \"kubernetes.io/projected/6f238afe-1ad4-4fca-a941-ebd07d08cf87-kube-api-access-cmjrd\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225134 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-token\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225387 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istiod-ca-cert\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225387 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225173 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-credential-socket\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225495 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-data\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225495 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-workload-certs\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225495 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-podinfo\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225649 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-workload-socket\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225649 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-envoy\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225649 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225627 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-credential-socket\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225821 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225728 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-data\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225901 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225877 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-workload-certs\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.225978 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-workload-socket\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.226042 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.225970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istiod-ca-cert\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.227982 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.227959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-envoy\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.228114 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.228095 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-podinfo\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.234732 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.234695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmjrd\" (UniqueName: \"kubernetes.io/projected/6f238afe-1ad4-4fca-a941-ebd07d08cf87-kube-api-access-cmjrd\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.236779 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.236757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-token\") pod \"e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.344245 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.344184 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:40.500683 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.500654 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg"] Jul 22 17:08:40.504446 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:08:40.504418 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f238afe_1ad4_4fca_a941_ebd07d08cf87.slice/crio-02a16d01934e48d7c7bd0dffabed70cae5205e1d1ca3037d12c3f0b4dfd984ff WatchSource:0}: Error finding container 02a16d01934e48d7c7bd0dffabed70cae5205e1d1ca3037d12c3f0b4dfd984ff: Status 404 returned error can't find the container with id 02a16d01934e48d7c7bd0dffabed70cae5205e1d1ca3037d12c3f0b4dfd984ff Jul 22 17:08:40.506442 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.506413 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:08:40.506499 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.506481 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:08:40.506539 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:40.506515 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:08:41.329119 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:41.329080 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" event={"ID":"6f238afe-1ad4-4fca-a941-ebd07d08cf87","Type":"ContainerStarted","Data":"3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874"} Jul 22 17:08:41.329491 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:41.329137 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" event={"ID":"6f238afe-1ad4-4fca-a941-ebd07d08cf87","Type":"ContainerStarted","Data":"02a16d01934e48d7c7bd0dffabed70cae5205e1d1ca3037d12c3f0b4dfd984ff"} Jul 22 17:08:41.344473 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:41.344442 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:41.349401 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:41.349377 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:41.352968 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:41.352926 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" podStartSLOduration=2.352912803 podStartE2EDuration="2.352912803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:08:41.350761842 +0000 UTC m=+1703.707158183" watchObservedRunningTime="2026-07-22 17:08:41.352912803 +0000 UTC m=+1703.709309122" Jul 22 17:08:42.333879 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:42.333851 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:42.335017 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:42.334991 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:08:49.696955 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:49.696920 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:08:57.424572 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:57.424539 2577 patch_prober.go:28] interesting pod/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": read tcp 10.133.0.2:44560->10.133.0.36:15021: read: connection reset by peer" start-of-body= Jul 22 17:08:57.424925 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:57.424609 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" podUID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": read tcp 10.133.0.2:44560->10.133.0.36:15021: read: connection reset by peer" Jul 22 17:08:58.410796 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:58.410760 2577 generic.go:356] "Generic (PLEG): container finished" podID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerID="3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874" exitCode=0 Jul 22 17:08:58.410960 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:58.410808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" event={"ID":"6f238afe-1ad4-4fca-a941-ebd07d08cf87","Type":"ContainerDied","Data":"3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874"} Jul 22 17:08:58.411190 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:58.411177 2577 scope.go:117] "RemoveContainer" containerID="3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874" Jul 22 17:08:58.412148 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:58.412122 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:08:58.412207 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:58.412188 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:08:58.412242 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:58.412219 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:08:59.417294 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:08:59.417263 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" event={"ID":"6f238afe-1ad4-4fca-a941-ebd07d08cf87","Type":"ContainerStarted","Data":"61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa"} Jul 22 17:09:00.345588 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:00.345169 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:09:00.345588 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:00.345290 2577 patch_prober.go:28] interesting pod/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 22 17:09:00.345588 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:00.345333 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" podUID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 22 17:09:01.345620 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:01.345572 2577 patch_prober.go:28] interesting pod/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 22 17:09:01.346129 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:01.345640 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" podUID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 22 17:09:02.345302 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:02.345261 2577 patch_prober.go:28] interesting pod/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 22 17:09:02.345583 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:02.345551 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" podUID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 22 17:09:03.348841 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:03.348811 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:09:03.349203 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:03.349055 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:09:03.349818 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:03.349800 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:09:21.620609 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:21.620575 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:09:23.493112 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:23.493079 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg"] Jul 22 17:09:23.493529 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:23.493277 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" podUID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerName="istio-proxy" containerID="cri-o://61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa" gracePeriod=30 Jul 22 17:09:24.736392 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.736356 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv"] Jul 22 17:09:24.739068 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.739047 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.746623 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.746350 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-af56de08-gw-openshift-default-dockercfg-gd9zg\"" Jul 22 17:09:24.776965 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.776918 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv"] Jul 22 17:09:24.799932 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.799901 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-workload-socket\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.800070 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.799945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s24z\" (UniqueName: \"kubernetes.io/projected/3f59719b-116d-4901-8672-992eb581ead1-kube-api-access-4s24z\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.800070 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.799979 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f59719b-116d-4901-8672-992eb581ead1-istio-token\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.800070 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.800015 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-istio-data\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.800070 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.800046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-credential-socket\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.800235 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.800089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f59719b-116d-4901-8672-992eb581ead1-istio-podinfo\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.800235 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.800116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f59719b-116d-4901-8672-992eb581ead1-istiod-ca-cert\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.800235 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.800179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-workload-certs\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.800332 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.800248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-istio-envoy\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.901472 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.901433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-workload-certs\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.901661 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.901495 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-istio-envoy\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.901661 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.901544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-workload-socket\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.901661 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.901570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s24z\" (UniqueName: \"kubernetes.io/projected/3f59719b-116d-4901-8672-992eb581ead1-kube-api-access-4s24z\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.901661 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.901596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f59719b-116d-4901-8672-992eb581ead1-istio-token\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.901661 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.901628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-istio-data\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.901661 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.901655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-credential-socket\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.901980 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.901696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f59719b-116d-4901-8672-992eb581ead1-istio-podinfo\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.901980 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.901727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f59719b-116d-4901-8672-992eb581ead1-istiod-ca-cert\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.902078 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.901973 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-workload-certs\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.902168 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.902146 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-istio-data\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.902271 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.902164 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-credential-socket\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.902324 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.902187 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-workload-socket\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.902504 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.902486 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f59719b-116d-4901-8672-992eb581ead1-istiod-ca-cert\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.903698 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.903675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-istio-envoy\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.904179 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.904162 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f59719b-116d-4901-8672-992eb581ead1-istio-podinfo\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.942569 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.942541 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s24z\" (UniqueName: \"kubernetes.io/projected/3f59719b-116d-4901-8672-992eb581ead1-kube-api-access-4s24z\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:24.958237 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:24.958209 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f59719b-116d-4901-8672-992eb581ead1-istio-token\") pod \"e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:25.050843 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:25.050758 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:25.229493 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:09:25.229459 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f59719b_116d_4901_8672_992eb581ead1.slice/crio-eaee9d1e150a1c401b874477f4c2f4b04e95402d9dec36052c40e0c1db37b9d7 WatchSource:0}: Error finding container eaee9d1e150a1c401b874477f4c2f4b04e95402d9dec36052c40e0c1db37b9d7: Status 404 returned error can't find the container with id eaee9d1e150a1c401b874477f4c2f4b04e95402d9dec36052c40e0c1db37b9d7 Jul 22 17:09:25.231619 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:25.231585 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:09:25.231691 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:25.231655 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:09:25.231691 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:25.231682 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:09:25.233156 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:25.233132 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv"] Jul 22 17:09:25.528520 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:25.528486 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" event={"ID":"3f59719b-116d-4901-8672-992eb581ead1","Type":"ContainerStarted","Data":"824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3"} Jul 22 17:09:25.528688 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:25.528541 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" event={"ID":"3f59719b-116d-4901-8672-992eb581ead1","Type":"ContainerStarted","Data":"eaee9d1e150a1c401b874477f4c2f4b04e95402d9dec36052c40e0c1db37b9d7"} Jul 22 17:09:25.557590 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:25.557536 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" podStartSLOduration=1.557518937 podStartE2EDuration="1.557518937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:09:25.557193286 +0000 UTC m=+1747.913589606" watchObservedRunningTime="2026-07-22 17:09:25.557518937 +0000 UTC m=+1747.913915255" Jul 22 17:09:26.051694 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:26.051660 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:26.056521 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:26.056496 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:26.539320 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:26.539279 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:26.540474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:26.540453 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:28.748114 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.748091 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:09:28.840806 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.840731 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-workload-certs\") pod \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " Jul 22 17:09:28.840806 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.840788 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmjrd\" (UniqueName: \"kubernetes.io/projected/6f238afe-1ad4-4fca-a941-ebd07d08cf87-kube-api-access-cmjrd\") pod \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " Jul 22 17:09:28.840806 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.840812 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-workload-socket\") pod \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " Jul 22 17:09:28.841050 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.840827 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-envoy\") pod \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " Jul 22 17:09:28.841050 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.840881 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-podinfo\") pod \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " Jul 22 17:09:28.841050 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.840903 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istiod-ca-cert\") pod \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " Jul 22 17:09:28.841050 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.840934 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-token\") pod \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " Jul 22 17:09:28.841050 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.840969 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-credential-socket\") pod \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " Jul 22 17:09:28.841050 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.840996 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-data\") pod \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\" (UID: \"6f238afe-1ad4-4fca-a941-ebd07d08cf87\") " Jul 22 17:09:28.841050 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.841015 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f238afe-1ad4-4fca-a941-ebd07d08cf87" (UID: "6f238afe-1ad4-4fca-a941-ebd07d08cf87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:09:28.841388 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.841115 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f238afe-1ad4-4fca-a941-ebd07d08cf87" (UID: "6f238afe-1ad4-4fca-a941-ebd07d08cf87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:09:28.841476 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.841456 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-workload-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:09:28.841538 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.841479 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-workload-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:09:28.841538 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.841497 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f238afe-1ad4-4fca-a941-ebd07d08cf87" (UID: "6f238afe-1ad4-4fca-a941-ebd07d08cf87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:09:28.841666 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.841632 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f238afe-1ad4-4fca-a941-ebd07d08cf87" (UID: "6f238afe-1ad4-4fca-a941-ebd07d08cf87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:09:28.842212 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.842186 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f238afe-1ad4-4fca-a941-ebd07d08cf87" (UID: "6f238afe-1ad4-4fca-a941-ebd07d08cf87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:09:28.843296 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.843270 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f238afe-1ad4-4fca-a941-ebd07d08cf87-kube-api-access-cmjrd" (OuterVolumeSpecName: "kube-api-access-cmjrd") pod "6f238afe-1ad4-4fca-a941-ebd07d08cf87" (UID: "6f238afe-1ad4-4fca-a941-ebd07d08cf87"). InnerVolumeSpecName "kube-api-access-cmjrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:09:28.843409 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.843359 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f238afe-1ad4-4fca-a941-ebd07d08cf87" (UID: "6f238afe-1ad4-4fca-a941-ebd07d08cf87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 17:09:28.843679 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.843662 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f238afe-1ad4-4fca-a941-ebd07d08cf87" (UID: "6f238afe-1ad4-4fca-a941-ebd07d08cf87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:09:28.843776 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.843662 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f238afe-1ad4-4fca-a941-ebd07d08cf87" (UID: "6f238afe-1ad4-4fca-a941-ebd07d08cf87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:09:28.942357 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.942324 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmjrd\" (UniqueName: \"kubernetes.io/projected/6f238afe-1ad4-4fca-a941-ebd07d08cf87-kube-api-access-cmjrd\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:09:28.942357 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.942357 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-envoy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:09:28.942499 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.942372 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-podinfo\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:09:28.942499 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.942385 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istiod-ca-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:09:28.942499 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.942397 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:09:28.942499 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.942409 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-credential-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:09:28.942499 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:28.942421 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f238afe-1ad4-4fca-a941-ebd07d08cf87-istio-data\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:09:29.554148 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:29.554116 2577 generic.go:356] "Generic (PLEG): container finished" podID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerID="61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa" exitCode=0 Jul 22 17:09:29.554344 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:29.554173 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" Jul 22 17:09:29.554344 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:29.554190 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" event={"ID":"6f238afe-1ad4-4fca-a941-ebd07d08cf87","Type":"ContainerDied","Data":"61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa"} Jul 22 17:09:29.554344 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:29.554232 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg" event={"ID":"6f238afe-1ad4-4fca-a941-ebd07d08cf87","Type":"ContainerDied","Data":"02a16d01934e48d7c7bd0dffabed70cae5205e1d1ca3037d12c3f0b4dfd984ff"} Jul 22 17:09:29.554344 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:29.554254 2577 scope.go:117] "RemoveContainer" containerID="61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa" Jul 22 17:09:29.563416 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:29.563398 2577 scope.go:117] "RemoveContainer" containerID="3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874" Jul 22 17:09:29.570719 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:29.570700 2577 scope.go:117] "RemoveContainer" containerID="61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa" Jul 22 17:09:29.570996 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:09:29.570977 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa\": container with ID starting with 61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa not found: ID does not exist" containerID="61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa" Jul 22 17:09:29.571067 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:29.571007 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa"} err="failed to get container status \"61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa\": rpc error: code = NotFound desc = could not find container \"61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa\": container with ID starting with 61abb24097de5bad3173e1bcc70e80d5b891c9c3ac49c70eb7a6df9b7b83e1aa not found: ID does not exist" Jul 22 17:09:29.571067 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:29.571031 2577 scope.go:117] "RemoveContainer" containerID="3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874" Jul 22 17:09:29.571245 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:09:29.571225 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874\": container with ID starting with 3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874 not found: ID does not exist" containerID="3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874" Jul 22 17:09:29.571285 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:29.571251 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874"} err="failed to get container status \"3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874\": rpc error: code = NotFound desc = could not find container \"3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874\": container with ID starting with 3d452521cea2252548b9d298634f6301a38d1f194ab61f4e469fa2d93c295874 not found: ID does not exist" Jul 22 17:09:29.585267 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:29.585246 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg"] Jul 22 17:09:29.589064 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:29.589044 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-76a9cf43-openshift-default-5c5b4fbbbf-kcfvg"] Jul 22 17:09:30.185915 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:30.185876 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" path="/var/lib/kubelet/pods/6f238afe-1ad4-4fca-a941-ebd07d08cf87/volumes" Jul 22 17:09:33.505501 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:33.505468 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:09:41.543209 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:41.541360 2577 patch_prober.go:28] interesting pod/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 22 17:09:41.543209 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:41.541441 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" podUID="3f59719b-116d-4901-8672-992eb581ead1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 22 17:09:41.607977 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:41.607936 2577 generic.go:356] "Generic (PLEG): container finished" podID="3f59719b-116d-4901-8672-992eb581ead1" containerID="824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3" exitCode=0 Jul 22 17:09:41.608136 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:41.608007 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" event={"ID":"3f59719b-116d-4901-8672-992eb581ead1","Type":"ContainerDied","Data":"824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3"} Jul 22 17:09:41.608435 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:41.608417 2577 scope.go:117] "RemoveContainer" containerID="824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3" Jul 22 17:09:41.609422 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:41.609395 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:09:41.609568 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:41.609457 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:09:41.609568 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:41.609486 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:09:42.614943 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:42.614913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" event={"ID":"3f59719b-116d-4901-8672-992eb581ead1","Type":"ContainerStarted","Data":"e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395"} Jul 22 17:09:43.051277 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:43.051246 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:43.051621 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:43.051596 2577 patch_prober.go:28] interesting pod/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 22 17:09:43.051816 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:43.051789 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" podUID="3f59719b-116d-4901-8672-992eb581ead1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 22 17:09:44.051112 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:44.051074 2577 patch_prober.go:28] interesting pod/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 22 17:09:44.051681 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:44.051648 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" podUID="3f59719b-116d-4901-8672-992eb581ead1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 22 17:09:45.052196 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:45.052150 2577 patch_prober.go:28] interesting pod/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 22 17:09:45.052660 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:45.052219 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" podUID="3f59719b-116d-4901-8672-992eb581ead1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 22 17:09:46.051763 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:46.051701 2577 patch_prober.go:28] interesting pod/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 22 17:09:46.052058 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:46.052023 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" podUID="3f59719b-116d-4901-8672-992eb581ead1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 22 17:09:47.055135 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:47.055109 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:47.055496 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:47.055374 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:09:47.056072 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:09:47.056055 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:10:05.703660 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:05.703629 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:10:07.326052 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:07.326019 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv"] Jul 22 17:10:07.326578 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:07.326272 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" podUID="3f59719b-116d-4901-8672-992eb581ead1" containerName="istio-proxy" containerID="cri-o://e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395" gracePeriod=30 Jul 22 17:10:08.108855 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.108818 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk"] Jul 22 17:10:08.109586 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.109566 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerName="istio-proxy" Jul 22 17:10:08.109638 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.109591 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerName="istio-proxy" Jul 22 17:10:08.109638 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.109603 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerName="istio-proxy" Jul 22 17:10:08.109638 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.109611 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerName="istio-proxy" Jul 22 17:10:08.109805 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.109792 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerName="istio-proxy" Jul 22 17:10:08.112570 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.112545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.115393 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.115370 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b408d9d9-openshift-default-dockercfg-bxb2p\"" Jul 22 17:10:08.125391 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.125367 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk"] Jul 22 17:10:08.271099 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.271066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d57883-a878-484e-b188-68e20a257eaf-istiod-ca-cert\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.271293 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.271108 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nplf\" (UniqueName: \"kubernetes.io/projected/a5d57883-a878-484e-b188-68e20a257eaf-kube-api-access-9nplf\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.271293 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.271145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-workload-socket\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.271293 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.271166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-istio-data\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.271293 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.271211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-workload-certs\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.271293 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.271239 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d57883-a878-484e-b188-68e20a257eaf-istio-podinfo\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.271293 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.271273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d57883-a878-484e-b188-68e20a257eaf-istio-token\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.271509 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.271324 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-istio-envoy\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.271509 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.271372 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-credential-socket\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.372662 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.372577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-workload-certs\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.372662 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.372623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d57883-a878-484e-b188-68e20a257eaf-istio-podinfo\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.373176 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.372663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d57883-a878-484e-b188-68e20a257eaf-istio-token\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.373176 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.372691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-istio-envoy\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.373176 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.372763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-credential-socket\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.373176 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.372832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d57883-a878-484e-b188-68e20a257eaf-istiod-ca-cert\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.373176 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.372864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nplf\" (UniqueName: \"kubernetes.io/projected/a5d57883-a878-484e-b188-68e20a257eaf-kube-api-access-9nplf\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.373176 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.372905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-workload-socket\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.373176 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.372929 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-istio-data\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.373176 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.373073 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-workload-certs\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.373589 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.373251 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-istio-data\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.373648 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.373608 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-credential-socket\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.373701 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.373682 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-workload-socket\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.373919 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.373762 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d57883-a878-484e-b188-68e20a257eaf-istiod-ca-cert\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.375479 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.375457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-istio-envoy\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.375802 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.375778 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d57883-a878-484e-b188-68e20a257eaf-istio-podinfo\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.382638 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.382607 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d57883-a878-484e-b188-68e20a257eaf-istio-token\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.383080 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.383060 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nplf\" (UniqueName: \"kubernetes.io/projected/a5d57883-a878-484e-b188-68e20a257eaf-kube-api-access-9nplf\") pod \"e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.423712 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.423688 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:08.573359 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.573329 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk"] Jul 22 17:10:08.576387 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:10:08.576361 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d57883_a878_484e_b188_68e20a257eaf.slice/crio-451535aeff60c9cf305099d665e11893ffc0948f0b1617189c32752747fc8826 WatchSource:0}: Error finding container 451535aeff60c9cf305099d665e11893ffc0948f0b1617189c32752747fc8826: Status 404 returned error can't find the container with id 451535aeff60c9cf305099d665e11893ffc0948f0b1617189c32752747fc8826 Jul 22 17:10:08.578404 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.578374 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:10:08.578462 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.578443 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:10:08.578499 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.578476 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:10:08.735642 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.735593 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" event={"ID":"a5d57883-a878-484e-b188-68e20a257eaf","Type":"ContainerStarted","Data":"37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa"} Jul 22 17:10:08.735817 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.735668 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" event={"ID":"a5d57883-a878-484e-b188-68e20a257eaf","Type":"ContainerStarted","Data":"451535aeff60c9cf305099d665e11893ffc0948f0b1617189c32752747fc8826"} Jul 22 17:10:08.757870 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:08.757814 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" podStartSLOduration=0.757793938 podStartE2EDuration="757.793938ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:10:08.756985635 +0000 UTC m=+1791.113381954" watchObservedRunningTime="2026-07-22 17:10:08.757793938 +0000 UTC m=+1791.114190257" Jul 22 17:10:09.424214 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:09.424177 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:09.428980 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:09.428958 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:09.742204 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:09.742117 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:09.743113 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:09.743088 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:12.584807 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.584785 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:10:12.612905 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.612850 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-istio-envoy\") pod \"3f59719b-116d-4901-8672-992eb581ead1\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " Jul 22 17:10:12.612905 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.612883 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-credential-socket\") pod \"3f59719b-116d-4901-8672-992eb581ead1\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " Jul 22 17:10:12.613085 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.612947 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-istio-data\") pod \"3f59719b-116d-4901-8672-992eb581ead1\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " Jul 22 17:10:12.613085 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.612988 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-workload-certs\") pod \"3f59719b-116d-4901-8672-992eb581ead1\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " Jul 22 17:10:12.613085 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.613038 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-workload-socket\") pod \"3f59719b-116d-4901-8672-992eb581ead1\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " Jul 22 17:10:12.613085 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.613063 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f59719b-116d-4901-8672-992eb581ead1-istiod-ca-cert\") pod \"3f59719b-116d-4901-8672-992eb581ead1\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " Jul 22 17:10:12.613283 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.613095 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s24z\" (UniqueName: \"kubernetes.io/projected/3f59719b-116d-4901-8672-992eb581ead1-kube-api-access-4s24z\") pod \"3f59719b-116d-4901-8672-992eb581ead1\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " Jul 22 17:10:12.613283 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.613131 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f59719b-116d-4901-8672-992eb581ead1-istio-podinfo\") pod \"3f59719b-116d-4901-8672-992eb581ead1\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " Jul 22 17:10:12.613283 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.613172 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f59719b-116d-4901-8672-992eb581ead1-istio-token\") pod \"3f59719b-116d-4901-8672-992eb581ead1\" (UID: \"3f59719b-116d-4901-8672-992eb581ead1\") " Jul 22 17:10:12.613283 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.613247 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3f59719b-116d-4901-8672-992eb581ead1" (UID: "3f59719b-116d-4901-8672-992eb581ead1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:10:12.613497 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.613469 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3f59719b-116d-4901-8672-992eb581ead1" (UID: "3f59719b-116d-4901-8672-992eb581ead1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:10:12.613617 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.613495 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-credential-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:12.613683 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.613631 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-istio-data" (OuterVolumeSpecName: "istio-data") pod "3f59719b-116d-4901-8672-992eb581ead1" (UID: "3f59719b-116d-4901-8672-992eb581ead1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:10:12.613761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.613704 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3f59719b-116d-4901-8672-992eb581ead1" (UID: "3f59719b-116d-4901-8672-992eb581ead1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:10:12.613819 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.613799 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f59719b-116d-4901-8672-992eb581ead1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3f59719b-116d-4901-8672-992eb581ead1" (UID: "3f59719b-116d-4901-8672-992eb581ead1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:10:12.615320 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.615287 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3f59719b-116d-4901-8672-992eb581ead1" (UID: "3f59719b-116d-4901-8672-992eb581ead1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:10:12.615600 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.615577 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f59719b-116d-4901-8672-992eb581ead1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3f59719b-116d-4901-8672-992eb581ead1" (UID: "3f59719b-116d-4901-8672-992eb581ead1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 17:10:12.615774 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.615627 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f59719b-116d-4901-8672-992eb581ead1-istio-token" (OuterVolumeSpecName: "istio-token") pod "3f59719b-116d-4901-8672-992eb581ead1" (UID: "3f59719b-116d-4901-8672-992eb581ead1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:10:12.615774 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.615637 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f59719b-116d-4901-8672-992eb581ead1-kube-api-access-4s24z" (OuterVolumeSpecName: "kube-api-access-4s24z") pod "3f59719b-116d-4901-8672-992eb581ead1" (UID: "3f59719b-116d-4901-8672-992eb581ead1"). InnerVolumeSpecName "kube-api-access-4s24z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:10:12.713969 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.713945 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-istio-data\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:12.713969 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.713968 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-workload-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:12.714108 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.713978 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-workload-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:12.714108 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.713986 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f59719b-116d-4901-8672-992eb581ead1-istiod-ca-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:12.714108 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.713994 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s24z\" (UniqueName: \"kubernetes.io/projected/3f59719b-116d-4901-8672-992eb581ead1-kube-api-access-4s24z\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:12.714108 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.714003 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f59719b-116d-4901-8672-992eb581ead1-istio-podinfo\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:12.714108 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.714011 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f59719b-116d-4901-8672-992eb581ead1-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:12.714108 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.714018 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f59719b-116d-4901-8672-992eb581ead1-istio-envoy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:12.757318 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.757293 2577 generic.go:356] "Generic (PLEG): container finished" podID="3f59719b-116d-4901-8672-992eb581ead1" containerID="e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395" exitCode=0 Jul 22 17:10:12.757430 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.757344 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" Jul 22 17:10:12.757430 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.757354 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" event={"ID":"3f59719b-116d-4901-8672-992eb581ead1","Type":"ContainerDied","Data":"e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395"} Jul 22 17:10:12.757430 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.757396 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv" event={"ID":"3f59719b-116d-4901-8672-992eb581ead1","Type":"ContainerDied","Data":"eaee9d1e150a1c401b874477f4c2f4b04e95402d9dec36052c40e0c1db37b9d7"} Jul 22 17:10:12.757430 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.757416 2577 scope.go:117] "RemoveContainer" containerID="e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395" Jul 22 17:10:12.767967 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.767943 2577 scope.go:117] "RemoveContainer" containerID="824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3" Jul 22 17:10:12.775727 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.775711 2577 scope.go:117] "RemoveContainer" containerID="e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395" Jul 22 17:10:12.776005 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:10:12.775976 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395\": container with ID starting with e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395 not found: ID does not exist" containerID="e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395" Jul 22 17:10:12.776068 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.776014 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395"} err="failed to get container status \"e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395\": rpc error: code = NotFound desc = could not find container \"e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395\": container with ID starting with e1a121acf40215956023921d7d9ff3d7f554680f9b8e060191da570946c5b395 not found: ID does not exist" Jul 22 17:10:12.776068 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.776030 2577 scope.go:117] "RemoveContainer" containerID="824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3" Jul 22 17:10:12.776239 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:10:12.776225 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3\": container with ID starting with 824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3 not found: ID does not exist" containerID="824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3" Jul 22 17:10:12.776278 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.776243 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3"} err="failed to get container status \"824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3\": rpc error: code = NotFound desc = could not find container \"824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3\": container with ID starting with 824646de793ad4a39d120c4c41017ba8f1f063756d997b07a9813ce16a259bb3 not found: ID does not exist" Jul 22 17:10:12.781206 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.781187 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv"] Jul 22 17:10:12.786704 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:12.786685 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-af56de08-gw-openshift-default-79d6c86b75-nwqlv"] Jul 22 17:10:14.185810 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:14.185777 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f59719b-116d-4901-8672-992eb581ead1" path="/var/lib/kubelet/pods/3f59719b-116d-4901-8672-992eb581ead1/volumes" Jul 22 17:10:16.797392 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:16.797358 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:10:24.744087 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:24.744049 2577 patch_prober.go:28] interesting pod/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 22 17:10:24.744493 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:24.744115 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" podUID="a5d57883-a878-484e-b188-68e20a257eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 22 17:10:24.813947 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:24.813916 2577 generic.go:356] "Generic (PLEG): container finished" podID="a5d57883-a878-484e-b188-68e20a257eaf" containerID="37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa" exitCode=0 Jul 22 17:10:24.814102 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:24.813997 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" event={"ID":"a5d57883-a878-484e-b188-68e20a257eaf","Type":"ContainerDied","Data":"37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa"} Jul 22 17:10:24.814391 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:24.814378 2577 scope.go:117] "RemoveContainer" containerID="37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa" Jul 22 17:10:24.815341 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:24.815314 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:10:24.815436 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:24.815376 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:10:24.815436 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:24.815409 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:10:25.820043 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:25.820007 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" event={"ID":"a5d57883-a878-484e-b188-68e20a257eaf","Type":"ContainerStarted","Data":"bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13"} Jul 22 17:10:26.424038 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:26.424006 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:26.424273 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:26.424252 2577 patch_prober.go:28] interesting pod/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 22 17:10:26.424363 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:26.424307 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" podUID="a5d57883-a878-484e-b188-68e20a257eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 22 17:10:27.425011 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:27.424970 2577 patch_prober.go:28] interesting pod/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 22 17:10:27.425398 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:27.425029 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" podUID="a5d57883-a878-484e-b188-68e20a257eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 22 17:10:28.424550 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:28.424514 2577 patch_prober.go:28] interesting pod/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 22 17:10:28.424723 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:28.424569 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" podUID="a5d57883-a878-484e-b188-68e20a257eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 22 17:10:29.424796 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:29.424760 2577 patch_prober.go:28] interesting pod/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 22 17:10:29.425179 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:29.424827 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" podUID="a5d57883-a878-484e-b188-68e20a257eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 22 17:10:30.428375 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:30.428346 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:30.428777 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:30.428605 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:30.429270 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:30.429253 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:49.500522 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:49.500486 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:10:52.235390 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:52.235353 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk"] Jul 22 17:10:52.235875 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:52.235598 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" podUID="a5d57883-a878-484e-b188-68e20a257eaf" containerName="istio-proxy" containerID="cri-o://bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13" gracePeriod=30 Jul 22 17:10:53.039577 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.039541 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8"] Jul 22 17:10:53.040077 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.040059 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f59719b-116d-4901-8672-992eb581ead1" containerName="istio-proxy" Jul 22 17:10:53.040140 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.040079 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f59719b-116d-4901-8672-992eb581ead1" containerName="istio-proxy" Jul 22 17:10:53.040192 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.040181 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f59719b-116d-4901-8672-992eb581ead1" containerName="istio-proxy" Jul 22 17:10:53.040226 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.040200 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f59719b-116d-4901-8672-992eb581ead1" containerName="istio-proxy" Jul 22 17:10:53.040226 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.040212 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f238afe-1ad4-4fca-a941-ebd07d08cf87" containerName="istio-proxy" Jul 22 17:10:53.042436 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.042394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.045242 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.045215 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f443994a-openshift-default-dockercfg-fj8pg\"" Jul 22 17:10:53.057075 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.057052 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8"] Jul 22 17:10:53.168587 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.168561 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-podinfo\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.168761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.168613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-workload-socket\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.168761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.168642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-token\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.168761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.168683 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-workload-certs\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.168761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.168704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-data\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.168761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.168720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-envoy\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.168761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.168757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqljf\" (UniqueName: \"kubernetes.io/projected/8e5c37be-c203-41bf-8e80-e2ef05a8b758-kube-api-access-sqljf\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.168996 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.168782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istiod-ca-cert\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.168996 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.168808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-credential-socket\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.269845 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.269809 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-workload-certs\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.269845 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.269844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-data\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.270347 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.269865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-envoy\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.270347 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.269993 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqljf\" (UniqueName: \"kubernetes.io/projected/8e5c37be-c203-41bf-8e80-e2ef05a8b758-kube-api-access-sqljf\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.270347 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.270039 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istiod-ca-cert\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.270347 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.270092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-credential-socket\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.270347 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.270175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-podinfo\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.270347 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.270177 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-data\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.270347 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.270238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-workload-socket\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.270347 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.270288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-workload-certs\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.270688 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.270353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-token\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.270688 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.270386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-credential-socket\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.270688 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.270533 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-workload-socket\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.270823 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.270808 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istiod-ca-cert\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.272473 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.272443 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-envoy\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.272620 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.272604 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-podinfo\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.278529 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.278509 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqljf\" (UniqueName: \"kubernetes.io/projected/8e5c37be-c203-41bf-8e80-e2ef05a8b758-kube-api-access-sqljf\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.279116 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.279097 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-token\") pod \"e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.353373 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.353302 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:53.486509 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.486475 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8"] Jul 22 17:10:53.490517 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:10:53.490488 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5c37be_c203_41bf_8e80_e2ef05a8b758.slice/crio-ef8e66359919dc60018ca44adf74f97c3e2c17202a81f51cebf0865e29f2687e WatchSource:0}: Error finding container ef8e66359919dc60018ca44adf74f97c3e2c17202a81f51cebf0865e29f2687e: Status 404 returned error can't find the container with id ef8e66359919dc60018ca44adf74f97c3e2c17202a81f51cebf0865e29f2687e Jul 22 17:10:53.492626 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.492580 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:10:53.492703 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.492659 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:10:53.492703 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.492686 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:10:53.952158 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.952111 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" event={"ID":"8e5c37be-c203-41bf-8e80-e2ef05a8b758","Type":"ContainerStarted","Data":"5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4"} Jul 22 17:10:53.952319 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.952234 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" event={"ID":"8e5c37be-c203-41bf-8e80-e2ef05a8b758","Type":"ContainerStarted","Data":"ef8e66359919dc60018ca44adf74f97c3e2c17202a81f51cebf0865e29f2687e"} Jul 22 17:10:53.977012 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:53.976962 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" podStartSLOduration=0.976941867 podStartE2EDuration="976.941867ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:10:53.975338473 +0000 UTC m=+1836.331734802" watchObservedRunningTime="2026-07-22 17:10:53.976941867 +0000 UTC m=+1836.333338187" Jul 22 17:10:54.354164 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:54.354077 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:54.359222 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:54.359197 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:54.959873 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:54.959844 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:54.960869 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:54.960848 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:10:57.486461 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.486441 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:57.607558 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.607479 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-istio-envoy\") pod \"a5d57883-a878-484e-b188-68e20a257eaf\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " Jul 22 17:10:57.607558 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.607533 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d57883-a878-484e-b188-68e20a257eaf-istio-token\") pod \"a5d57883-a878-484e-b188-68e20a257eaf\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " Jul 22 17:10:57.607794 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.607566 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-workload-socket\") pod \"a5d57883-a878-484e-b188-68e20a257eaf\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " Jul 22 17:10:57.607794 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.607597 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-workload-certs\") pod \"a5d57883-a878-484e-b188-68e20a257eaf\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " Jul 22 17:10:57.607794 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.607774 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d57883-a878-484e-b188-68e20a257eaf-istiod-ca-cert\") pod \"a5d57883-a878-484e-b188-68e20a257eaf\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " Jul 22 17:10:57.607969 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.607819 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nplf\" (UniqueName: \"kubernetes.io/projected/a5d57883-a878-484e-b188-68e20a257eaf-kube-api-access-9nplf\") pod \"a5d57883-a878-484e-b188-68e20a257eaf\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " Jul 22 17:10:57.607969 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.607845 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d57883-a878-484e-b188-68e20a257eaf-istio-podinfo\") pod \"a5d57883-a878-484e-b188-68e20a257eaf\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " Jul 22 17:10:57.607969 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.607876 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-istio-data\") pod \"a5d57883-a878-484e-b188-68e20a257eaf\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " Jul 22 17:10:57.607969 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.607915 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-credential-socket\") pod \"a5d57883-a878-484e-b188-68e20a257eaf\" (UID: \"a5d57883-a878-484e-b188-68e20a257eaf\") " Jul 22 17:10:57.607969 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.607929 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5d57883-a878-484e-b188-68e20a257eaf" (UID: "a5d57883-a878-484e-b188-68e20a257eaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:10:57.608219 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.607951 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5d57883-a878-484e-b188-68e20a257eaf" (UID: "a5d57883-a878-484e-b188-68e20a257eaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:10:57.608219 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.608202 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d57883-a878-484e-b188-68e20a257eaf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5d57883-a878-484e-b188-68e20a257eaf" (UID: "a5d57883-a878-484e-b188-68e20a257eaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:10:57.608338 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.608216 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-workload-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:57.608338 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.608234 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-workload-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:57.608438 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.608398 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5d57883-a878-484e-b188-68e20a257eaf" (UID: "a5d57883-a878-484e-b188-68e20a257eaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:10:57.608711 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.608690 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5d57883-a878-484e-b188-68e20a257eaf" (UID: "a5d57883-a878-484e-b188-68e20a257eaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:10:57.609670 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.609645 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5d57883-a878-484e-b188-68e20a257eaf" (UID: "a5d57883-a878-484e-b188-68e20a257eaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:10:57.609919 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.609898 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5d57883-a878-484e-b188-68e20a257eaf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5d57883-a878-484e-b188-68e20a257eaf" (UID: "a5d57883-a878-484e-b188-68e20a257eaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 17:10:57.610161 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.610142 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d57883-a878-484e-b188-68e20a257eaf-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5d57883-a878-484e-b188-68e20a257eaf" (UID: "a5d57883-a878-484e-b188-68e20a257eaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:10:57.610294 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.610272 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d57883-a878-484e-b188-68e20a257eaf-kube-api-access-9nplf" (OuterVolumeSpecName: "kube-api-access-9nplf") pod "a5d57883-a878-484e-b188-68e20a257eaf" (UID: "a5d57883-a878-484e-b188-68e20a257eaf"). InnerVolumeSpecName "kube-api-access-9nplf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:10:57.708652 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.708626 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-istio-envoy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:57.708652 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.708648 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d57883-a878-484e-b188-68e20a257eaf-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:57.708821 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.708660 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d57883-a878-484e-b188-68e20a257eaf-istiod-ca-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:57.708821 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.708673 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nplf\" (UniqueName: \"kubernetes.io/projected/a5d57883-a878-484e-b188-68e20a257eaf-kube-api-access-9nplf\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:57.708821 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.708685 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d57883-a878-484e-b188-68e20a257eaf-istio-podinfo\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:57.708821 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.708697 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-istio-data\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:57.708821 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.708708 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d57883-a878-484e-b188-68e20a257eaf-credential-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:10:57.975841 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.975810 2577 generic.go:356] "Generic (PLEG): container finished" podID="a5d57883-a878-484e-b188-68e20a257eaf" containerID="bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13" exitCode=0 Jul 22 17:10:57.975982 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.975842 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" event={"ID":"a5d57883-a878-484e-b188-68e20a257eaf","Type":"ContainerDied","Data":"bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13"} Jul 22 17:10:57.975982 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.975898 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" event={"ID":"a5d57883-a878-484e-b188-68e20a257eaf","Type":"ContainerDied","Data":"451535aeff60c9cf305099d665e11893ffc0948f0b1617189c32752747fc8826"} Jul 22 17:10:57.975982 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.975855 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk" Jul 22 17:10:57.976143 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.975927 2577 scope.go:117] "RemoveContainer" containerID="bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13" Jul 22 17:10:57.984950 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.984935 2577 scope.go:117] "RemoveContainer" containerID="37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa" Jul 22 17:10:57.992554 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.992538 2577 scope.go:117] "RemoveContainer" containerID="bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13" Jul 22 17:10:57.992813 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:10:57.992793 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13\": container with ID starting with bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13 not found: ID does not exist" containerID="bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13" Jul 22 17:10:57.992871 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.992821 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13"} err="failed to get container status \"bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13\": rpc error: code = NotFound desc = could not find container \"bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13\": container with ID starting with bb4565d04dff7a66dc79ce8eab6f2190c1322162649e8a4b4c3dc0109177ae13 not found: ID does not exist" Jul 22 17:10:57.992871 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.992838 2577 scope.go:117] "RemoveContainer" containerID="37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa" Jul 22 17:10:57.993069 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:10:57.993053 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa\": container with ID starting with 37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa not found: ID does not exist" containerID="37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa" Jul 22 17:10:57.993110 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:57.993075 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa"} err="failed to get container status \"37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa\": rpc error: code = NotFound desc = could not find container \"37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa\": container with ID starting with 37295626ddaf32077e7ff9980d70de929f2334ecca4d69f61498cc54d0e2bfaa not found: ID does not exist" Jul 22 17:10:58.002663 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:58.002641 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk"] Jul 22 17:10:58.006957 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:58.006937 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b408d9d9-openshift-default-5bd9774579-hz8rk"] Jul 22 17:10:58.185725 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:10:58.185697 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5d57883-a878-484e-b188-68e20a257eaf" path="/var/lib/kubelet/pods/a5d57883-a878-484e-b188-68e20a257eaf/volumes" Jul 22 17:11:03.104114 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:03.104081 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:11:10.331132 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:10.331010 2577 patch_prober.go:28] interesting pod/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:36432->10.133.0.39:15021: read: connection reset by peer" start-of-body= Jul 22 17:11:10.331132 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:10.331084 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:36432->10.133.0.39:15021: read: connection reset by peer" Jul 22 17:11:11.036479 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:11.036444 2577 generic.go:356] "Generic (PLEG): container finished" podID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerID="5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4" exitCode=0 Jul 22 17:11:11.036645 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:11.036518 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" event={"ID":"8e5c37be-c203-41bf-8e80-e2ef05a8b758","Type":"ContainerDied","Data":"5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4"} Jul 22 17:11:11.036965 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:11.036944 2577 scope.go:117] "RemoveContainer" containerID="5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4" Jul 22 17:11:11.038076 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:11.038024 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:11:11.038170 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:11.038131 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:11:11.038210 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:11.038174 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:11:12.046454 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:12.046406 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" event={"ID":"8e5c37be-c203-41bf-8e80-e2ef05a8b758","Type":"ContainerStarted","Data":"8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b"} Jul 22 17:11:12.355260 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:12.354682 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:11:12.355260 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:12.354838 2577 patch_prober.go:28] interesting pod/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 22 17:11:12.355260 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:12.354923 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 22 17:11:13.354728 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:13.354684 2577 patch_prober.go:28] interesting pod/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 22 17:11:13.355225 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:13.354782 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 22 17:11:14.354203 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:14.354164 2577 patch_prober.go:28] interesting pod/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 22 17:11:14.354486 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:14.354457 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 22 17:11:15.353763 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:15.353706 2577 patch_prober.go:28] interesting pod/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 22 17:11:15.354262 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:15.353804 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 22 17:11:16.358398 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:16.358368 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:11:16.358864 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:16.358700 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:11:16.359552 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:16.359534 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:11:33.601996 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:33.601957 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:11:35.437561 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:35.437530 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8"] Jul 22 17:11:35.438020 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:35.437792 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerName="istio-proxy" containerID="cri-o://8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b" gracePeriod=30 Jul 22 17:11:40.698111 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.698082 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:11:40.873347 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873273 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqljf\" (UniqueName: \"kubernetes.io/projected/8e5c37be-c203-41bf-8e80-e2ef05a8b758-kube-api-access-sqljf\") pod \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " Jul 22 17:11:40.873347 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873309 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istiod-ca-cert\") pod \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " Jul 22 17:11:40.873540 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873351 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-envoy\") pod \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " Jul 22 17:11:40.873540 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873383 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-podinfo\") pod \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " Jul 22 17:11:40.873540 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873408 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-workload-socket\") pod \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " Jul 22 17:11:40.873540 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873450 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-workload-certs\") pod \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " Jul 22 17:11:40.873540 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873489 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-credential-socket\") pod \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " Jul 22 17:11:40.873540 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873516 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-token\") pod \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " Jul 22 17:11:40.873540 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873537 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-data\") pod \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\" (UID: \"8e5c37be-c203-41bf-8e80-e2ef05a8b758\") " Jul 22 17:11:40.873946 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873766 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e5c37be-c203-41bf-8e80-e2ef05a8b758" (UID: "8e5c37be-c203-41bf-8e80-e2ef05a8b758"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:11:40.873946 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873783 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e5c37be-c203-41bf-8e80-e2ef05a8b758" (UID: "8e5c37be-c203-41bf-8e80-e2ef05a8b758"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:11:40.873946 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873926 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e5c37be-c203-41bf-8e80-e2ef05a8b758" (UID: "8e5c37be-c203-41bf-8e80-e2ef05a8b758"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:11:40.874075 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.873989 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e5c37be-c203-41bf-8e80-e2ef05a8b758" (UID: "8e5c37be-c203-41bf-8e80-e2ef05a8b758"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:11:40.874785 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.874732 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e5c37be-c203-41bf-8e80-e2ef05a8b758" (UID: "8e5c37be-c203-41bf-8e80-e2ef05a8b758"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:11:40.875708 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.875685 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e5c37be-c203-41bf-8e80-e2ef05a8b758" (UID: "8e5c37be-c203-41bf-8e80-e2ef05a8b758"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:11:40.875836 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.875725 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e5c37be-c203-41bf-8e80-e2ef05a8b758" (UID: "8e5c37be-c203-41bf-8e80-e2ef05a8b758"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 17:11:40.875879 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.875864 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e5c37be-c203-41bf-8e80-e2ef05a8b758" (UID: "8e5c37be-c203-41bf-8e80-e2ef05a8b758"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:11:40.875914 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.875871 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5c37be-c203-41bf-8e80-e2ef05a8b758-kube-api-access-sqljf" (OuterVolumeSpecName: "kube-api-access-sqljf") pod "8e5c37be-c203-41bf-8e80-e2ef05a8b758" (UID: "8e5c37be-c203-41bf-8e80-e2ef05a8b758"). InnerVolumeSpecName "kube-api-access-sqljf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:11:40.974598 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.974560 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-workload-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:11:40.974598 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.974600 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-credential-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:11:40.974812 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.974619 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:11:40.974812 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.974635 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-data\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:11:40.974812 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.974649 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqljf\" (UniqueName: \"kubernetes.io/projected/8e5c37be-c203-41bf-8e80-e2ef05a8b758-kube-api-access-sqljf\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:11:40.974812 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.974661 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istiod-ca-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:11:40.974812 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.974674 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-envoy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:11:40.974812 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.974686 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5c37be-c203-41bf-8e80-e2ef05a8b758-istio-podinfo\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:11:40.974812 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:40.974698 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c37be-c203-41bf-8e80-e2ef05a8b758-workload-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:11:41.177137 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:41.177103 2577 generic.go:356] "Generic (PLEG): container finished" podID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerID="8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b" exitCode=0 Jul 22 17:11:41.177318 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:41.177154 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" Jul 22 17:11:41.177318 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:41.177169 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" event={"ID":"8e5c37be-c203-41bf-8e80-e2ef05a8b758","Type":"ContainerDied","Data":"8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b"} Jul 22 17:11:41.177318 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:41.177226 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8" event={"ID":"8e5c37be-c203-41bf-8e80-e2ef05a8b758","Type":"ContainerDied","Data":"ef8e66359919dc60018ca44adf74f97c3e2c17202a81f51cebf0865e29f2687e"} Jul 22 17:11:41.177318 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:41.177248 2577 scope.go:117] "RemoveContainer" containerID="8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b" Jul 22 17:11:41.190869 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:41.190845 2577 scope.go:117] "RemoveContainer" containerID="5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4" Jul 22 17:11:41.198307 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:41.198290 2577 scope.go:117] "RemoveContainer" containerID="8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b" Jul 22 17:11:41.198539 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:11:41.198518 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b\": container with ID starting with 8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b not found: ID does not exist" containerID="8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b" Jul 22 17:11:41.198581 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:41.198546 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b"} err="failed to get container status \"8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b\": rpc error: code = NotFound desc = could not find container \"8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b\": container with ID starting with 8062fd3e6c12070b94bc8859ea0406be87d553327c1c5d94bcf145706dc8a68b not found: ID does not exist" Jul 22 17:11:41.198581 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:41.198563 2577 scope.go:117] "RemoveContainer" containerID="5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4" Jul 22 17:11:41.198793 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:11:41.198774 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4\": container with ID starting with 5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4 not found: ID does not exist" containerID="5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4" Jul 22 17:11:41.198837 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:41.198799 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4"} err="failed to get container status \"5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4\": rpc error: code = NotFound desc = could not find container \"5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4\": container with ID starting with 5a528b70faed0dbfc903432b77d956d228d8e9693c5682c2e174440c72e0c5b4 not found: ID does not exist" Jul 22 17:11:41.204415 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:41.204390 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8"] Jul 22 17:11:41.214639 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:41.214615 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f443994a-openshift-default-5d9cc6b4f5-ntzr8"] Jul 22 17:11:42.186044 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:11:42.186013 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" path="/var/lib/kubelet/pods/8e5c37be-c203-41bf-8e80-e2ef05a8b758/volumes" Jul 22 17:13:04.541084 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:04.541047 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:13:15.249959 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:15.249879 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:13:20.664439 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.664409 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz"] Jul 22 17:13:20.664969 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.664949 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d57883-a878-484e-b188-68e20a257eaf" containerName="istio-proxy" Jul 22 17:13:20.665045 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.664973 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d57883-a878-484e-b188-68e20a257eaf" containerName="istio-proxy" Jul 22 17:13:20.665045 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.664985 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerName="istio-proxy" Jul 22 17:13:20.665045 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.664995 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerName="istio-proxy" Jul 22 17:13:20.665045 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.665008 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f59719b-116d-4901-8672-992eb581ead1" containerName="istio-proxy" Jul 22 17:13:20.665045 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.665016 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f59719b-116d-4901-8672-992eb581ead1" containerName="istio-proxy" Jul 22 17:13:20.665045 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.665030 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerName="istio-proxy" Jul 22 17:13:20.665045 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.665039 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerName="istio-proxy" Jul 22 17:13:20.665413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.665048 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d57883-a878-484e-b188-68e20a257eaf" containerName="istio-proxy" Jul 22 17:13:20.665413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.665059 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d57883-a878-484e-b188-68e20a257eaf" containerName="istio-proxy" Jul 22 17:13:20.665413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.665155 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d57883-a878-484e-b188-68e20a257eaf" containerName="istio-proxy" Jul 22 17:13:20.665413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.665165 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d57883-a878-484e-b188-68e20a257eaf" containerName="istio-proxy" Jul 22 17:13:20.665413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.665179 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerName="istio-proxy" Jul 22 17:13:20.665413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.665191 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5c37be-c203-41bf-8e80-e2ef05a8b758" containerName="istio-proxy" Jul 22 17:13:20.668353 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.668334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.672406 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.672382 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-51445c0e-openshift-default-dockercfg-qv4nn\"" Jul 22 17:13:20.718698 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.718657 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz"] Jul 22 17:13:20.790343 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.790299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-workload-socket\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.790343 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.790347 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-podinfo\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.790550 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.790397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab119cc-8b30-427e-8e59-70c5b6648c9d-istiod-ca-cert\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.790550 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.790434 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-workload-certs\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.790550 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.790458 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-data\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.790550 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.790489 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-envoy\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.790550 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.790537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-credential-socket\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.790713 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.790558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-token\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.790713 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.790652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fktpk\" (UniqueName: \"kubernetes.io/projected/fab119cc-8b30-427e-8e59-70c5b6648c9d-kube-api-access-fktpk\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.891885 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.891841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-workload-certs\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.891885 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.891892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-data\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.892145 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.891938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-envoy\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.892145 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.891959 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-credential-socket\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.892145 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.891983 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-token\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.892145 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.892097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fktpk\" (UniqueName: \"kubernetes.io/projected/fab119cc-8b30-427e-8e59-70c5b6648c9d-kube-api-access-fktpk\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.892145 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.892128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-workload-socket\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.892407 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.892156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-podinfo\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.892407 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.892178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab119cc-8b30-427e-8e59-70c5b6648c9d-istiod-ca-cert\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.892407 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.892201 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-workload-certs\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.892407 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.892286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-data\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.892407 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.892330 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-credential-socket\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.892632 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.892568 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-workload-socket\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.892920 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.892891 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab119cc-8b30-427e-8e59-70c5b6648c9d-istiod-ca-cert\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.894479 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.894460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-envoy\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.894674 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.894657 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-podinfo\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.904366 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.904341 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fktpk\" (UniqueName: \"kubernetes.io/projected/fab119cc-8b30-427e-8e59-70c5b6648c9d-kube-api-access-fktpk\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.905050 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.905031 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-token\") pod \"e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:20.981010 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:20.980913 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:21.128713 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:21.128687 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz"] Jul 22 17:13:21.131303 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:13:21.131273 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab119cc_8b30_427e_8e59_70c5b6648c9d.slice/crio-0cdb37217aeb696f449ad1ef98ebd9b9b77904589616442dccde556e7dd892ae WatchSource:0}: Error finding container 0cdb37217aeb696f449ad1ef98ebd9b9b77904589616442dccde556e7dd892ae: Status 404 returned error can't find the container with id 0cdb37217aeb696f449ad1ef98ebd9b9b77904589616442dccde556e7dd892ae Jul 22 17:13:21.133485 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:21.133453 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:13:21.133576 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:21.133551 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:13:21.133623 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:21.133583 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:13:21.607333 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:21.607295 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" event={"ID":"fab119cc-8b30-427e-8e59-70c5b6648c9d","Type":"ContainerStarted","Data":"daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e"} Jul 22 17:13:21.607539 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:21.607345 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" event={"ID":"fab119cc-8b30-427e-8e59-70c5b6648c9d","Type":"ContainerStarted","Data":"0cdb37217aeb696f449ad1ef98ebd9b9b77904589616442dccde556e7dd892ae"} Jul 22 17:13:21.640046 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:21.639988 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" podStartSLOduration=1.639971252 podStartE2EDuration="1.639971252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:13:21.63952128 +0000 UTC m=+1983.995917624" watchObservedRunningTime="2026-07-22 17:13:21.639971252 +0000 UTC m=+1983.996367570" Jul 22 17:13:21.981597 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:21.981546 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:22.986424 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:22.986391 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:23.616394 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:23.616366 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:23.617295 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:23.617278 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:29.861142 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:29.861103 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:13:37.689253 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:37.689214 2577 generic.go:356] "Generic (PLEG): container finished" podID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerID="daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e" exitCode=0 Jul 22 17:13:37.689692 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:37.689271 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" event={"ID":"fab119cc-8b30-427e-8e59-70c5b6648c9d","Type":"ContainerDied","Data":"daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e"} Jul 22 17:13:37.689692 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:37.689637 2577 scope.go:117] "RemoveContainer" containerID="daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e" Jul 22 17:13:37.690633 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:37.690606 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:13:37.690705 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:37.690672 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:13:37.690765 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:37.690703 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:13:38.617153 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:38.617117 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:38.695408 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:38.695371 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" event={"ID":"fab119cc-8b30-427e-8e59-70c5b6648c9d","Type":"ContainerStarted","Data":"38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff"} Jul 22 17:13:38.981241 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:38.981201 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:38.982487 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:38.982458 2577 patch_prober.go:28] interesting pod/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 22 17:13:38.982623 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:38.982521 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 22 17:13:39.981524 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:39.981490 2577 patch_prober.go:28] interesting pod/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 22 17:13:39.982051 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:39.981567 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 22 17:13:40.982258 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:40.982218 2577 patch_prober.go:28] interesting pod/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 22 17:13:40.982653 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:40.982285 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 22 17:13:41.981474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:41.981433 2577 patch_prober.go:28] interesting pod/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 22 17:13:41.981655 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:41.981496 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 22 17:13:42.981694 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:42.981647 2577 patch_prober.go:28] interesting pod/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 22 17:13:42.982186 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:42.981714 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 22 17:13:43.985671 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:43.985641 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:43.986047 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:43.985960 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:43.986730 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:43.986713 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:13:49.960056 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:49.960014 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:13:59.355307 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:59.355270 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz"] Jul 22 17:13:59.355706 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:13:59.355503 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerName="istio-proxy" containerID="cri-o://38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff" gracePeriod=30 Jul 22 17:14:04.613109 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.613084 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:14:04.674411 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.674387 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-workload-socket\") pod \"fab119cc-8b30-427e-8e59-70c5b6648c9d\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " Jul 22 17:14:04.674563 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.674429 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-workload-certs\") pod \"fab119cc-8b30-427e-8e59-70c5b6648c9d\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " Jul 22 17:14:04.674563 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.674449 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab119cc-8b30-427e-8e59-70c5b6648c9d-istiod-ca-cert\") pod \"fab119cc-8b30-427e-8e59-70c5b6648c9d\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " Jul 22 17:14:04.674563 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.674485 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-envoy\") pod \"fab119cc-8b30-427e-8e59-70c5b6648c9d\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " Jul 22 17:14:04.674563 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.674502 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-data\") pod \"fab119cc-8b30-427e-8e59-70c5b6648c9d\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " Jul 22 17:14:04.674563 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.674523 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-credential-socket\") pod \"fab119cc-8b30-427e-8e59-70c5b6648c9d\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " Jul 22 17:14:04.674830 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.674564 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-podinfo\") pod \"fab119cc-8b30-427e-8e59-70c5b6648c9d\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " Jul 22 17:14:04.674830 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.674684 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fab119cc-8b30-427e-8e59-70c5b6648c9d" (UID: "fab119cc-8b30-427e-8e59-70c5b6648c9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:14:04.674830 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.674697 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fab119cc-8b30-427e-8e59-70c5b6648c9d" (UID: "fab119cc-8b30-427e-8e59-70c5b6648c9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:14:04.674830 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.674694 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-token\") pod \"fab119cc-8b30-427e-8e59-70c5b6648c9d\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " Jul 22 17:14:04.674830 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.674781 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fktpk\" (UniqueName: \"kubernetes.io/projected/fab119cc-8b30-427e-8e59-70c5b6648c9d-kube-api-access-fktpk\") pod \"fab119cc-8b30-427e-8e59-70c5b6648c9d\" (UID: \"fab119cc-8b30-427e-8e59-70c5b6648c9d\") " Jul 22 17:14:04.675079 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.674920 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fab119cc-8b30-427e-8e59-70c5b6648c9d" (UID: "fab119cc-8b30-427e-8e59-70c5b6648c9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:14:04.675079 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.675028 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab119cc-8b30-427e-8e59-70c5b6648c9d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fab119cc-8b30-427e-8e59-70c5b6648c9d" (UID: "fab119cc-8b30-427e-8e59-70c5b6648c9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:14:04.675181 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.675106 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-workload-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:14:04.675181 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.675121 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-workload-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:14:04.675181 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.675134 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab119cc-8b30-427e-8e59-70c5b6648c9d-istiod-ca-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:14:04.675181 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.675143 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-credential-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:14:04.675506 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.675475 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-data" (OuterVolumeSpecName: "istio-data") pod "fab119cc-8b30-427e-8e59-70c5b6648c9d" (UID: "fab119cc-8b30-427e-8e59-70c5b6648c9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:14:04.676925 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.676901 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab119cc-8b30-427e-8e59-70c5b6648c9d-kube-api-access-fktpk" (OuterVolumeSpecName: "kube-api-access-fktpk") pod "fab119cc-8b30-427e-8e59-70c5b6648c9d" (UID: "fab119cc-8b30-427e-8e59-70c5b6648c9d"). InnerVolumeSpecName "kube-api-access-fktpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:14:04.676925 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.676912 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-token" (OuterVolumeSpecName: "istio-token") pod "fab119cc-8b30-427e-8e59-70c5b6648c9d" (UID: "fab119cc-8b30-427e-8e59-70c5b6648c9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:14:04.677055 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.676918 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fab119cc-8b30-427e-8e59-70c5b6648c9d" (UID: "fab119cc-8b30-427e-8e59-70c5b6648c9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:14:04.677455 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.677442 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fab119cc-8b30-427e-8e59-70c5b6648c9d" (UID: "fab119cc-8b30-427e-8e59-70c5b6648c9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 17:14:04.775715 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.775675 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fktpk\" (UniqueName: \"kubernetes.io/projected/fab119cc-8b30-427e-8e59-70c5b6648c9d-kube-api-access-fktpk\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:14:04.775715 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.775708 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-envoy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:14:04.775715 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.775718 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-data\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:14:04.775715 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.775726 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-podinfo\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:14:04.776012 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.775776 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab119cc-8b30-427e-8e59-70c5b6648c9d-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:14:04.817210 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.817177 2577 generic.go:356] "Generic (PLEG): container finished" podID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerID="38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff" exitCode=0 Jul 22 17:14:04.817356 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.817231 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" Jul 22 17:14:04.817356 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.817246 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" event={"ID":"fab119cc-8b30-427e-8e59-70c5b6648c9d","Type":"ContainerDied","Data":"38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff"} Jul 22 17:14:04.817356 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.817289 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz" event={"ID":"fab119cc-8b30-427e-8e59-70c5b6648c9d","Type":"ContainerDied","Data":"0cdb37217aeb696f449ad1ef98ebd9b9b77904589616442dccde556e7dd892ae"} Jul 22 17:14:04.817356 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.817311 2577 scope.go:117] "RemoveContainer" containerID="38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff" Jul 22 17:14:04.826443 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.826428 2577 scope.go:117] "RemoveContainer" containerID="daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e" Jul 22 17:14:04.839695 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.839668 2577 scope.go:117] "RemoveContainer" containerID="38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff" Jul 22 17:14:04.840067 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:14:04.840045 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff\": container with ID starting with 38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff not found: ID does not exist" containerID="38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff" Jul 22 17:14:04.840147 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.840075 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff"} err="failed to get container status \"38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff\": rpc error: code = NotFound desc = could not find container \"38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff\": container with ID starting with 38c4e5d85a465f139d1b4831373fa447b6e33fb0679334183bcdef3d9170acff not found: ID does not exist" Jul 22 17:14:04.840147 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.840105 2577 scope.go:117] "RemoveContainer" containerID="daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e" Jul 22 17:14:04.840443 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:14:04.840419 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e\": container with ID starting with daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e not found: ID does not exist" containerID="daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e" Jul 22 17:14:04.840507 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.840454 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e"} err="failed to get container status \"daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e\": rpc error: code = NotFound desc = could not find container \"daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e\": container with ID starting with daf4054a5cf169187d37f2acc26d832ee07a01736b3b2c54a3023c760dfdff8e not found: ID does not exist" Jul 22 17:14:04.852525 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.852502 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz"] Jul 22 17:14:04.861216 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:04.861197 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-51445c0e-openshift-default-8654f65f8c-rscxz"] Jul 22 17:14:06.187381 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:06.187349 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" path="/var/lib/kubelet/pods/fab119cc-8b30-427e-8e59-70c5b6648c9d/volumes" Jul 22 17:14:19.851623 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.851591 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p"] Jul 22 17:14:19.852006 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.851954 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerName="istio-proxy" Jul 22 17:14:19.852006 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.851966 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerName="istio-proxy" Jul 22 17:14:19.852006 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.851975 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerName="istio-proxy" Jul 22 17:14:19.852006 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.851980 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerName="istio-proxy" Jul 22 17:14:19.852139 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.852063 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerName="istio-proxy" Jul 22 17:14:19.856413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.856389 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:19.859102 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.859082 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c1b75e3f-openshift-default-dockercfg-8g86f\"" Jul 22 17:14:19.869254 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.869228 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p"] Jul 22 17:14:19.905520 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.905483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-workload-socket\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:19.905520 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.905522 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndr2\" (UniqueName: \"kubernetes.io/projected/5dab7674-f535-4acf-a4d2-931ec2943030-kube-api-access-zndr2\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:19.905724 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.905546 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dab7674-f535-4acf-a4d2-931ec2943030-istio-podinfo\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:19.905724 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.905624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-credential-socket\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:19.905724 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.905667 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-istio-envoy\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:19.905724 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.905686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dab7674-f535-4acf-a4d2-931ec2943030-istio-token\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:19.905864 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.905722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-workload-certs\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:19.905864 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.905793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dab7674-f535-4acf-a4d2-931ec2943030-istiod-ca-cert\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:19.905864 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:19.905814 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-istio-data\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.006795 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.006760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-workload-certs\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.007009 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.006806 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dab7674-f535-4acf-a4d2-931ec2943030-istiod-ca-cert\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.007009 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.006830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-istio-data\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.007009 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.006861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-workload-socket\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.007009 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.006890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndr2\" (UniqueName: \"kubernetes.io/projected/5dab7674-f535-4acf-a4d2-931ec2943030-kube-api-access-zndr2\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.007009 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.006925 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dab7674-f535-4acf-a4d2-931ec2943030-istio-podinfo\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.007009 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.006953 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-credential-socket\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.007009 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.007000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-istio-envoy\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.007385 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.007022 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dab7674-f535-4acf-a4d2-931ec2943030-istio-token\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.007385 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.007190 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-workload-certs\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.007690 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.007642 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-credential-socket\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.007824 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.007776 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-workload-socket\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.008056 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.008031 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dab7674-f535-4acf-a4d2-931ec2943030-istiod-ca-cert\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.008168 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.008151 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-istio-data\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.009665 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.009642 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-istio-envoy\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.009854 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.009838 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dab7674-f535-4acf-a4d2-931ec2943030-istio-podinfo\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.016309 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.016285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndr2\" (UniqueName: \"kubernetes.io/projected/5dab7674-f535-4acf-a4d2-931ec2943030-kube-api-access-zndr2\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.016434 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.016287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dab7674-f535-4acf-a4d2-931ec2943030-istio-token\") pod \"e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.168292 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.168257 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:20.312910 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.312881 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p"] Jul 22 17:14:20.314918 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:14:20.314889 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dab7674_f535_4acf_a4d2_931ec2943030.slice/crio-7079763d355235ff0259732d1260315a927c5ecabc01d6d3586c63fa449a68dd WatchSource:0}: Error finding container 7079763d355235ff0259732d1260315a927c5ecabc01d6d3586c63fa449a68dd: Status 404 returned error can't find the container with id 7079763d355235ff0259732d1260315a927c5ecabc01d6d3586c63fa449a68dd Jul 22 17:14:20.317122 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.317089 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:14:20.317201 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.317159 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:14:20.317201 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.317186 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:14:20.889105 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.889074 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" event={"ID":"5dab7674-f535-4acf-a4d2-931ec2943030","Type":"ContainerStarted","Data":"288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9"} Jul 22 17:14:20.889105 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.889124 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" event={"ID":"5dab7674-f535-4acf-a4d2-931ec2943030","Type":"ContainerStarted","Data":"7079763d355235ff0259732d1260315a927c5ecabc01d6d3586c63fa449a68dd"} Jul 22 17:14:20.913367 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:20.913312 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" podStartSLOduration=1.913291261 podStartE2EDuration="1.913291261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:14:20.912295991 +0000 UTC m=+2043.268692309" watchObservedRunningTime="2026-07-22 17:14:20.913291261 +0000 UTC m=+2043.269687581" Jul 22 17:14:21.169328 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:21.169288 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:21.174383 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:21.174356 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:21.893472 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:21.893444 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:21.894624 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:21.894601 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:29.611863 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:29.611826 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:14:37.051665 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:37.051620 2577 patch_prober.go:28] interesting pod/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:56314->10.133.0.41:15021: read: connection reset by peer" start-of-body= Jul 22 17:14:37.052360 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:37.051683 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:56314->10.133.0.41:15021: read: connection reset by peer" Jul 22 17:14:37.969060 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:37.969026 2577 generic.go:356] "Generic (PLEG): container finished" podID="5dab7674-f535-4acf-a4d2-931ec2943030" containerID="288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9" exitCode=0 Jul 22 17:14:37.969223 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:37.969101 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" event={"ID":"5dab7674-f535-4acf-a4d2-931ec2943030","Type":"ContainerDied","Data":"288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9"} Jul 22 17:14:37.969493 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:37.969480 2577 scope.go:117] "RemoveContainer" containerID="288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9" Jul 22 17:14:37.970531 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:37.970506 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:14:37.970585 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:37.970573 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:14:37.970620 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:37.970604 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:14:38.975033 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:38.974999 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" event={"ID":"5dab7674-f535-4acf-a4d2-931ec2943030","Type":"ContainerStarted","Data":"defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310"} Jul 22 17:14:39.169158 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:39.169122 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:39.169464 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:39.169443 2577 patch_prober.go:28] interesting pod/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 17:14:39.169517 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:39.169492 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 17:14:40.168975 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:40.168935 2577 patch_prober.go:28] interesting pod/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 17:14:40.169370 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:40.168993 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 17:14:41.168887 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:41.168847 2577 patch_prober.go:28] interesting pod/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 17:14:41.169067 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:41.168907 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 17:14:42.168930 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:42.168887 2577 patch_prober.go:28] interesting pod/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 17:14:42.169114 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:42.168955 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 17:14:43.172410 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:43.172381 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:43.172898 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:43.172763 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:43.173376 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:43.173359 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:14:56.592893 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.592856 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6"] Jul 22 17:14:56.593390 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.593273 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab119cc-8b30-427e-8e59-70c5b6648c9d" containerName="istio-proxy" Jul 22 17:14:56.595357 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.595323 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.598482 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.598455 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c1b75e3f\"/\"default-dockercfg-cmjpx\"" Jul 22 17:14:56.598482 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.598468 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c1b75e3f\"/\"kube-root-ca.crt\"" Jul 22 17:14:56.598700 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.598533 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c1b75e3f\"/\"openshift-service-ca.crt\"" Jul 22 17:14:56.598807 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.598698 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c1b75e3f\"/\"e2e-lifecycle-model-c1b75e3f-kserve-self-signed-certs\"" Jul 22 17:14:56.607209 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.607188 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6"] Jul 22 17:14:56.737542 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.737501 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-model-cache\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.737805 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.737560 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk7hd\" (UniqueName: \"kubernetes.io/projected/a2d23607-013c-42d1-bddf-30b83e9eff87-kube-api-access-mk7hd\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.737805 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.737679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-tmp-dir\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.737805 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.737722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-dshm\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.737994 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.737806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d23607-013c-42d1-bddf-30b83e9eff87-tls-certs\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.737994 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.737829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-home\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.839203 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.839167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-model-cache\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.839413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.839219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk7hd\" (UniqueName: \"kubernetes.io/projected/a2d23607-013c-42d1-bddf-30b83e9eff87-kube-api-access-mk7hd\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.839413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.839269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-tmp-dir\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.839413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.839294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-dshm\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.839413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.839322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d23607-013c-42d1-bddf-30b83e9eff87-tls-certs\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.839637 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.839479 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-home\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.839785 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.839731 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-model-cache\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.839867 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.839823 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-tmp-dir\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.839867 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.839839 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-home\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.841844 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.841824 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-dshm\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.842096 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.842078 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d23607-013c-42d1-bddf-30b83e9eff87-tls-certs\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.847496 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.847443 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk7hd\" (UniqueName: \"kubernetes.io/projected/a2d23607-013c-42d1-bddf-30b83e9eff87-kube-api-access-mk7hd\") pod \"e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:56.906217 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:56.906179 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:14:57.049090 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:57.049060 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6"] Jul 22 17:14:57.051109 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:14:57.051081 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2d23607_013c_42d1_bddf_30b83e9eff87.slice/crio-585ba81a4a3e0308b2890ff50093fb9dcbd596693a0b5ebb380bca0d8a0ccf4b WatchSource:0}: Error finding container 585ba81a4a3e0308b2890ff50093fb9dcbd596693a0b5ebb380bca0d8a0ccf4b: Status 404 returned error can't find the container with id 585ba81a4a3e0308b2890ff50093fb9dcbd596693a0b5ebb380bca0d8a0ccf4b Jul 22 17:14:57.062975 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:57.062948 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" event={"ID":"a2d23607-013c-42d1-bddf-30b83e9eff87","Type":"ContainerStarted","Data":"585ba81a4a3e0308b2890ff50093fb9dcbd596693a0b5ebb380bca0d8a0ccf4b"} Jul 22 17:14:58.069057 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:58.069022 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" event={"ID":"a2d23607-013c-42d1-bddf-30b83e9eff87","Type":"ContainerStarted","Data":"8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030"} Jul 22 17:14:58.092198 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:14:58.092140 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" podStartSLOduration=2.09212351 podStartE2EDuration="2.09212351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:14:58.085134965 +0000 UTC m=+2080.441531283" watchObservedRunningTime="2026-07-22 17:14:58.09212351 +0000 UTC m=+2080.448519827" Jul 22 17:15:00.194792 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:15:00.194756 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:15:06.906549 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:15:06.906506 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:15:06.906986 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:15:06.906563 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:15:06.919066 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:15:06.919044 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:15:07.122348 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:15:07.122313 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:15:17.894678 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:15:17.894633 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:15:35.397882 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:15:35.397838 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:15:45.889677 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:15:45.889642 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:15:48.945770 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:15:48.945718 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6"] Jul 22 17:16:03.966940 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:03.966869 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" podUID="a2d23607-013c-42d1-bddf-30b83e9eff87" containerName="main" containerID="cri-o://8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030" gracePeriod=45 Jul 22 17:16:04.212465 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.212442 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:16:04.319006 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.318919 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-dshm\") pod \"a2d23607-013c-42d1-bddf-30b83e9eff87\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " Jul 22 17:16:04.319006 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.318962 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk7hd\" (UniqueName: \"kubernetes.io/projected/a2d23607-013c-42d1-bddf-30b83e9eff87-kube-api-access-mk7hd\") pod \"a2d23607-013c-42d1-bddf-30b83e9eff87\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " Jul 22 17:16:04.319006 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.318988 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-model-cache\") pod \"a2d23607-013c-42d1-bddf-30b83e9eff87\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " Jul 22 17:16:04.319282 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.319029 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-tmp-dir\") pod \"a2d23607-013c-42d1-bddf-30b83e9eff87\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " Jul 22 17:16:04.319282 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.319093 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-home\") pod \"a2d23607-013c-42d1-bddf-30b83e9eff87\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " Jul 22 17:16:04.319282 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.319126 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d23607-013c-42d1-bddf-30b83e9eff87-tls-certs\") pod \"a2d23607-013c-42d1-bddf-30b83e9eff87\" (UID: \"a2d23607-013c-42d1-bddf-30b83e9eff87\") " Jul 22 17:16:04.319474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.319334 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2d23607-013c-42d1-bddf-30b83e9eff87" (UID: "a2d23607-013c-42d1-bddf-30b83e9eff87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:16:04.319474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.319346 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2d23607-013c-42d1-bddf-30b83e9eff87" (UID: "a2d23607-013c-42d1-bddf-30b83e9eff87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:16:04.319474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.319358 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-home" (OuterVolumeSpecName: "home") pod "a2d23607-013c-42d1-bddf-30b83e9eff87" (UID: "a2d23607-013c-42d1-bddf-30b83e9eff87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:16:04.319474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.319436 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-tmp-dir\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:04.321146 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.321118 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-dshm" (OuterVolumeSpecName: "dshm") pod "a2d23607-013c-42d1-bddf-30b83e9eff87" (UID: "a2d23607-013c-42d1-bddf-30b83e9eff87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:16:04.321276 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.321251 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d23607-013c-42d1-bddf-30b83e9eff87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2d23607-013c-42d1-bddf-30b83e9eff87" (UID: "a2d23607-013c-42d1-bddf-30b83e9eff87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:16:04.321335 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.321259 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d23607-013c-42d1-bddf-30b83e9eff87-kube-api-access-mk7hd" (OuterVolumeSpecName: "kube-api-access-mk7hd") pod "a2d23607-013c-42d1-bddf-30b83e9eff87" (UID: "a2d23607-013c-42d1-bddf-30b83e9eff87"). InnerVolumeSpecName "kube-api-access-mk7hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:16:04.367855 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.367824 2577 generic.go:356] "Generic (PLEG): container finished" podID="a2d23607-013c-42d1-bddf-30b83e9eff87" containerID="8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030" exitCode=0 Jul 22 17:16:04.368009 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.367869 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" Jul 22 17:16:04.368009 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.367871 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" event={"ID":"a2d23607-013c-42d1-bddf-30b83e9eff87","Type":"ContainerDied","Data":"8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030"} Jul 22 17:16:04.368009 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.367961 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6" event={"ID":"a2d23607-013c-42d1-bddf-30b83e9eff87","Type":"ContainerDied","Data":"585ba81a4a3e0308b2890ff50093fb9dcbd596693a0b5ebb380bca0d8a0ccf4b"} Jul 22 17:16:04.368009 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.367981 2577 scope.go:117] "RemoveContainer" containerID="8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030" Jul 22 17:16:04.376581 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.376564 2577 scope.go:117] "RemoveContainer" containerID="8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030" Jul 22 17:16:04.376877 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:16:04.376854 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030\": container with ID starting with 8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030 not found: ID does not exist" containerID="8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030" Jul 22 17:16:04.376961 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.376885 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030"} err="failed to get container status \"8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030\": rpc error: code = NotFound desc = could not find container \"8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030\": container with ID starting with 8d83aa225352326d273361a419e292234e5235b4731e5e09f67428e520052030 not found: ID does not exist" Jul 22 17:16:04.388312 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.388288 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6"] Jul 22 17:16:04.391453 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.391434 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c1b75e3f/e2e-lifecycle-model-c1b75e3f-kserve-5b659b84cf-8mxs6"] Jul 22 17:16:04.420491 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.420467 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-home\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:04.420588 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.420494 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d23607-013c-42d1-bddf-30b83e9eff87-tls-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:04.420588 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.420505 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-dshm\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:04.420588 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.420514 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk7hd\" (UniqueName: \"kubernetes.io/projected/a2d23607-013c-42d1-bddf-30b83e9eff87-kube-api-access-mk7hd\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:04.420588 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:04.420524 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d23607-013c-42d1-bddf-30b83e9eff87-model-cache\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:06.186039 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:06.186005 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d23607-013c-42d1-bddf-30b83e9eff87" path="/var/lib/kubelet/pods/a2d23607-013c-42d1-bddf-30b83e9eff87/volumes" Jul 22 17:16:31.936357 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:31.936319 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p"] Jul 22 17:16:31.936909 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:31.936606 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" containerName="istio-proxy" containerID="cri-o://defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310" gracePeriod=30 Jul 22 17:16:36.002353 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:36.002318 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:16:37.177004 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.176977 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:16:37.212533 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.212454 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-workload-socket\") pod \"5dab7674-f535-4acf-a4d2-931ec2943030\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " Jul 22 17:16:37.212533 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.212503 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-credential-socket\") pod \"5dab7674-f535-4acf-a4d2-931ec2943030\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " Jul 22 17:16:37.212761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.212538 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dab7674-f535-4acf-a4d2-931ec2943030-istio-podinfo\") pod \"5dab7674-f535-4acf-a4d2-931ec2943030\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " Jul 22 17:16:37.212761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.212564 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dab7674-f535-4acf-a4d2-931ec2943030-istio-token\") pod \"5dab7674-f535-4acf-a4d2-931ec2943030\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " Jul 22 17:16:37.212761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.212599 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zndr2\" (UniqueName: \"kubernetes.io/projected/5dab7674-f535-4acf-a4d2-931ec2943030-kube-api-access-zndr2\") pod \"5dab7674-f535-4acf-a4d2-931ec2943030\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " Jul 22 17:16:37.212761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.212630 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-workload-certs\") pod \"5dab7674-f535-4acf-a4d2-931ec2943030\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " Jul 22 17:16:37.212761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.212686 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-istio-envoy\") pod \"5dab7674-f535-4acf-a4d2-931ec2943030\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " Jul 22 17:16:37.212761 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.212715 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dab7674-f535-4acf-a4d2-931ec2943030-istiod-ca-cert\") pod \"5dab7674-f535-4acf-a4d2-931ec2943030\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " Jul 22 17:16:37.213088 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.212775 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-istio-data\") pod \"5dab7674-f535-4acf-a4d2-931ec2943030\" (UID: \"5dab7674-f535-4acf-a4d2-931ec2943030\") " Jul 22 17:16:37.213088 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.212776 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5dab7674-f535-4acf-a4d2-931ec2943030" (UID: "5dab7674-f535-4acf-a4d2-931ec2943030"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:16:37.213088 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.212788 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5dab7674-f535-4acf-a4d2-931ec2943030" (UID: "5dab7674-f535-4acf-a4d2-931ec2943030"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:16:37.213088 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.213036 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-workload-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:37.213088 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.213054 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-credential-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:37.213332 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.213148 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5dab7674-f535-4acf-a4d2-931ec2943030" (UID: "5dab7674-f535-4acf-a4d2-931ec2943030"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:16:37.213332 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.213172 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dab7674-f535-4acf-a4d2-931ec2943030-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5dab7674-f535-4acf-a4d2-931ec2943030" (UID: "5dab7674-f535-4acf-a4d2-931ec2943030"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:16:37.213913 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.213883 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-istio-data" (OuterVolumeSpecName: "istio-data") pod "5dab7674-f535-4acf-a4d2-931ec2943030" (UID: "5dab7674-f535-4acf-a4d2-931ec2943030"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:16:37.214930 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.214892 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dab7674-f535-4acf-a4d2-931ec2943030-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5dab7674-f535-4acf-a4d2-931ec2943030" (UID: "5dab7674-f535-4acf-a4d2-931ec2943030"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 17:16:37.215044 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.214972 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5dab7674-f535-4acf-a4d2-931ec2943030" (UID: "5dab7674-f535-4acf-a4d2-931ec2943030"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:16:37.215229 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.215205 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dab7674-f535-4acf-a4d2-931ec2943030-istio-token" (OuterVolumeSpecName: "istio-token") pod "5dab7674-f535-4acf-a4d2-931ec2943030" (UID: "5dab7674-f535-4acf-a4d2-931ec2943030"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:16:37.215322 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.215229 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dab7674-f535-4acf-a4d2-931ec2943030-kube-api-access-zndr2" (OuterVolumeSpecName: "kube-api-access-zndr2") pod "5dab7674-f535-4acf-a4d2-931ec2943030" (UID: "5dab7674-f535-4acf-a4d2-931ec2943030"). InnerVolumeSpecName "kube-api-access-zndr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:16:37.313667 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.313633 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dab7674-f535-4acf-a4d2-931ec2943030-istio-podinfo\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:37.313667 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.313661 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dab7674-f535-4acf-a4d2-931ec2943030-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:37.313667 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.313673 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zndr2\" (UniqueName: \"kubernetes.io/projected/5dab7674-f535-4acf-a4d2-931ec2943030-kube-api-access-zndr2\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:37.313930 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.313686 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-workload-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:37.313930 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.313698 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-istio-envoy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:37.313930 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.313708 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dab7674-f535-4acf-a4d2-931ec2943030-istiod-ca-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:37.313930 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.313718 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dab7674-f535-4acf-a4d2-931ec2943030-istio-data\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:16:37.513596 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.513502 2577 generic.go:356] "Generic (PLEG): container finished" podID="5dab7674-f535-4acf-a4d2-931ec2943030" containerID="defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310" exitCode=0 Jul 22 17:16:37.513596 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.513567 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" Jul 22 17:16:37.513865 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.513587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" event={"ID":"5dab7674-f535-4acf-a4d2-931ec2943030","Type":"ContainerDied","Data":"defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310"} Jul 22 17:16:37.513865 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.513657 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p" event={"ID":"5dab7674-f535-4acf-a4d2-931ec2943030","Type":"ContainerDied","Data":"7079763d355235ff0259732d1260315a927c5ecabc01d6d3586c63fa449a68dd"} Jul 22 17:16:37.513865 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.513687 2577 scope.go:117] "RemoveContainer" containerID="defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310" Jul 22 17:16:37.523484 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.523460 2577 scope.go:117] "RemoveContainer" containerID="288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9" Jul 22 17:16:37.532140 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.532115 2577 scope.go:117] "RemoveContainer" containerID="defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310" Jul 22 17:16:37.532399 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:16:37.532380 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310\": container with ID starting with defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310 not found: ID does not exist" containerID="defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310" Jul 22 17:16:37.532493 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.532406 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310"} err="failed to get container status \"defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310\": rpc error: code = NotFound desc = could not find container \"defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310\": container with ID starting with defb5fbcb45dc1b0def191a78941afa39bac41acf3d2526cc713ab5ff0f8e310 not found: ID does not exist" Jul 22 17:16:37.532493 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.532424 2577 scope.go:117] "RemoveContainer" containerID="288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9" Jul 22 17:16:37.532639 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:16:37.532620 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9\": container with ID starting with 288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9 not found: ID does not exist" containerID="288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9" Jul 22 17:16:37.532704 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.532642 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9"} err="failed to get container status \"288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9\": rpc error: code = NotFound desc = could not find container \"288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9\": container with ID starting with 288f993d9c1af313a426ef6b76f18e72682ca742f266523090aa283c121c31e9 not found: ID does not exist" Jul 22 17:16:37.539335 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.539309 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p"] Jul 22 17:16:37.543511 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:37.543484 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c1b75e3f-openshift-default-f4f89749f-ctd6p"] Jul 22 17:16:38.185381 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:38.185348 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" path="/var/lib/kubelet/pods/5dab7674-f535-4acf-a4d2-931ec2943030/volumes" Jul 22 17:16:41.798189 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:41.798150 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:16:45.498886 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:16:45.498849 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:17:28.465022 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.464982 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l"] Jul 22 17:17:28.465578 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.465518 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" containerName="istio-proxy" Jul 22 17:17:28.465578 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.465538 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" containerName="istio-proxy" Jul 22 17:17:28.465578 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.465570 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d23607-013c-42d1-bddf-30b83e9eff87" containerName="main" Jul 22 17:17:28.465578 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.465579 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d23607-013c-42d1-bddf-30b83e9eff87" containerName="main" Jul 22 17:17:28.465825 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.465665 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d23607-013c-42d1-bddf-30b83e9eff87" containerName="main" Jul 22 17:17:28.465825 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.465682 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" containerName="istio-proxy" Jul 22 17:17:28.465825 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.465691 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" containerName="istio-proxy" Jul 22 17:17:28.468627 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.468608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.471087 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.471064 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-668a76-openshift-default-dockercfg-vh7t5\"" Jul 22 17:17:28.490928 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.490902 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l"] Jul 22 17:17:28.551776 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.551723 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-workload-certs\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.552065 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.552034 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e78813-8b78-4dc0-a1b6-830856865824-istiod-ca-cert\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.552201 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.552110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e78813-8b78-4dc0-a1b6-830856865824-istio-token\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.552201 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.552157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-credential-socket\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.552358 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.552237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-workload-socket\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.552358 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.552278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djcsg\" (UniqueName: \"kubernetes.io/projected/31e78813-8b78-4dc0-a1b6-830856865824-kube-api-access-djcsg\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.552358 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.552319 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-istio-data\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.552358 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.552344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e78813-8b78-4dc0-a1b6-830856865824-istio-podinfo\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.552567 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.552393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-istio-envoy\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.652990 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.652961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-istio-envoy\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.653182 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.653026 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-workload-certs\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.653182 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.653057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e78813-8b78-4dc0-a1b6-830856865824-istiod-ca-cert\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.653182 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.653097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e78813-8b78-4dc0-a1b6-830856865824-istio-token\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.653182 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.653126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-credential-socket\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.653182 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.653154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-workload-socket\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.653453 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.653188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djcsg\" (UniqueName: \"kubernetes.io/projected/31e78813-8b78-4dc0-a1b6-830856865824-kube-api-access-djcsg\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.653453 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.653215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-istio-data\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.653453 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.653243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e78813-8b78-4dc0-a1b6-830856865824-istio-podinfo\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.653616 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.653468 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-workload-certs\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.653616 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.653501 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-credential-socket\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.653803 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.653781 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-workload-socket\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.653983 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.653962 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e78813-8b78-4dc0-a1b6-830856865824-istiod-ca-cert\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.654161 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.654138 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-istio-data\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.655438 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.655419 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-istio-envoy\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.655714 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.655697 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e78813-8b78-4dc0-a1b6-830856865824-istio-podinfo\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.666029 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.666003 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e78813-8b78-4dc0-a1b6-830856865824-istio-token\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.668507 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.668485 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djcsg\" (UniqueName: \"kubernetes.io/projected/31e78813-8b78-4dc0-a1b6-830856865824-kube-api-access-djcsg\") pod \"e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.778271 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.778190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:28.928316 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.928269 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l"] Jul 22 17:17:28.931324 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:17:28.931294 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e78813_8b78_4dc0_a1b6_830856865824.slice/crio-6ef98f34298f03a1364e137872cf96706015522d637fbee2044645a9921fb8f0 WatchSource:0}: Error finding container 6ef98f34298f03a1364e137872cf96706015522d637fbee2044645a9921fb8f0: Status 404 returned error can't find the container with id 6ef98f34298f03a1364e137872cf96706015522d637fbee2044645a9921fb8f0 Jul 22 17:17:28.933563 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.933526 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:17:28.933630 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.933607 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:17:28.933673 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:28.933648 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:17:29.747748 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:29.747708 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" event={"ID":"31e78813-8b78-4dc0-a1b6-830856865824","Type":"ContainerStarted","Data":"4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a"} Jul 22 17:17:29.748141 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:29.747786 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" event={"ID":"31e78813-8b78-4dc0-a1b6-830856865824","Type":"ContainerStarted","Data":"6ef98f34298f03a1364e137872cf96706015522d637fbee2044645a9921fb8f0"} Jul 22 17:17:29.778609 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:29.778581 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:29.783194 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:29.783174 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:29.788025 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:29.787984 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" podStartSLOduration=1.787970346 podStartE2EDuration="1.787970346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:17:29.786113592 +0000 UTC m=+2232.142509910" watchObservedRunningTime="2026-07-22 17:17:29.787970346 +0000 UTC m=+2232.144366663" Jul 22 17:17:30.752560 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:30.752528 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:30.753403 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:30.753382 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:36.926581 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:36.926545 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz"] Jul 22 17:17:36.927142 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:36.927123 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" containerName="istio-proxy" Jul 22 17:17:36.927142 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:36.927143 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dab7674-f535-4acf-a4d2-931ec2943030" containerName="istio-proxy" Jul 22 17:17:36.930600 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:36.930579 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:36.932657 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:36.932633 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f854fd-openshift-default-dockercfg-pcs2w\"" Jul 22 17:17:36.943550 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:36.943526 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz"] Jul 22 17:17:37.021429 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.021396 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-envoy\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.021607 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.021442 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-workload-certs\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.021607 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.021490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-credential-socket\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.021607 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.021536 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67gb8\" (UniqueName: \"kubernetes.io/projected/02fbc22f-f9c3-482a-ac5e-e13745edd384-kube-api-access-67gb8\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.021816 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.021647 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02fbc22f-f9c3-482a-ac5e-e13745edd384-istiod-ca-cert\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.021816 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.021702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-token\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.021816 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.021732 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-podinfo\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.021816 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.021788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-workload-socket\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.021974 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.021826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-data\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.122658 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.122618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67gb8\" (UniqueName: \"kubernetes.io/projected/02fbc22f-f9c3-482a-ac5e-e13745edd384-kube-api-access-67gb8\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.122877 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.122694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02fbc22f-f9c3-482a-ac5e-e13745edd384-istiod-ca-cert\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.122877 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.122731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-token\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.122995 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.122882 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-podinfo\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.122995 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.122929 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-workload-socket\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.122995 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.122954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-data\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.123136 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.123021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-envoy\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.123136 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.123046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-workload-certs\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.123136 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.123093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-credential-socket\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.123363 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.123340 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-data\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.123449 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.123436 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-credential-socket\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.123514 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.123490 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-workload-certs\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.123565 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.123517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02fbc22f-f9c3-482a-ac5e-e13745edd384-istiod-ca-cert\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.123565 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.123546 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-workload-socket\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.125283 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.125262 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-envoy\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.125563 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.125541 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-podinfo\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.131297 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.131268 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-token\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.131392 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.131296 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67gb8\" (UniqueName: \"kubernetes.io/projected/02fbc22f-f9c3-482a-ac5e-e13745edd384-kube-api-access-67gb8\") pod \"e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.193541 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.193461 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:17:37.243214 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.243182 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:37.384274 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.384250 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz"] Jul 22 17:17:37.386679 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:17:37.386650 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fbc22f_f9c3_482a_ac5e_e13745edd384.slice/crio-80e49de7415b36b74f0c3cbeb62ed995e2ec2a488be7d41d8af3ec8be2497171 WatchSource:0}: Error finding container 80e49de7415b36b74f0c3cbeb62ed995e2ec2a488be7d41d8af3ec8be2497171: Status 404 returned error can't find the container with id 80e49de7415b36b74f0c3cbeb62ed995e2ec2a488be7d41d8af3ec8be2497171 Jul 22 17:17:37.388712 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.388681 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:17:37.388821 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.388764 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:17:37.388821 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.388795 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:17:37.787162 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.787070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" event={"ID":"02fbc22f-f9c3-482a-ac5e-e13745edd384","Type":"ContainerStarted","Data":"96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164"} Jul 22 17:17:37.787162 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.787131 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" event={"ID":"02fbc22f-f9c3-482a-ac5e-e13745edd384","Type":"ContainerStarted","Data":"80e49de7415b36b74f0c3cbeb62ed995e2ec2a488be7d41d8af3ec8be2497171"} Jul 22 17:17:37.809555 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:37.809505 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" podStartSLOduration=1.809490152 podStartE2EDuration="1.809490152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:17:37.806371752 +0000 UTC m=+2240.162768070" watchObservedRunningTime="2026-07-22 17:17:37.809490152 +0000 UTC m=+2240.165886470" Jul 22 17:17:38.244125 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:38.244093 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:38.252909 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:38.252875 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:38.799469 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:38.799430 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:38.800331 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:38.800311 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:45.754086 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:45.754041 2577 patch_prober.go:28] interesting pod/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 22 17:17:45.754631 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:45.754146 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 22 17:17:45.833188 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:45.833155 2577 generic.go:356] "Generic (PLEG): container finished" podID="31e78813-8b78-4dc0-a1b6-830856865824" containerID="4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a" exitCode=0 Jul 22 17:17:45.833188 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:45.833195 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" event={"ID":"31e78813-8b78-4dc0-a1b6-830856865824","Type":"ContainerDied","Data":"4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a"} Jul 22 17:17:45.833618 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:45.833604 2577 scope.go:117] "RemoveContainer" containerID="4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a" Jul 22 17:17:45.834710 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:45.834682 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:17:45.834804 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:45.834789 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:17:45.834865 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:45.834838 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:17:46.842285 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:46.842235 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" event={"ID":"31e78813-8b78-4dc0-a1b6-830856865824","Type":"ContainerStarted","Data":"24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b"} Jul 22 17:17:47.779131 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:47.779091 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:47.779341 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:47.779245 2577 patch_prober.go:28] interesting pod/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 22 17:17:47.779341 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:47.779309 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 22 17:17:48.778988 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:48.778949 2577 patch_prober.go:28] interesting pod/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 22 17:17:48.779597 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:48.779565 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 22 17:17:49.778996 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:49.778960 2577 patch_prober.go:28] interesting pod/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 22 17:17:49.779400 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:49.779019 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 22 17:17:50.778963 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:50.778930 2577 patch_prober.go:28] interesting pod/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 22 17:17:50.779158 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:50.778992 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 22 17:17:51.782666 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:51.782634 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:51.783128 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:51.782925 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:51.783574 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:51.783560 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:17:52.879226 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:52.879142 2577 generic.go:356] "Generic (PLEG): container finished" podID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerID="96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164" exitCode=0 Jul 22 17:17:52.879226 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:52.879217 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" event={"ID":"02fbc22f-f9c3-482a-ac5e-e13745edd384","Type":"ContainerDied","Data":"96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164"} Jul 22 17:17:52.879670 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:52.879655 2577 scope.go:117] "RemoveContainer" containerID="96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164" Jul 22 17:17:52.880708 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:52.880680 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:17:52.880792 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:52.880779 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:17:52.880845 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:52.880829 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:17:53.299775 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:53.299721 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:17:53.799542 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:53.799513 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:53.888385 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:53.888350 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" event={"ID":"02fbc22f-f9c3-482a-ac5e-e13745edd384","Type":"ContainerStarted","Data":"750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db"} Jul 22 17:17:54.244785 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:54.244751 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:54.245200 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:54.245178 2577 patch_prober.go:28] interesting pod/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 22 17:17:54.245375 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:54.245351 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 22 17:17:55.244134 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:55.244097 2577 patch_prober.go:28] interesting pod/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 22 17:17:55.245080 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:55.245046 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 22 17:17:56.244544 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:56.244500 2577 patch_prober.go:28] interesting pod/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 22 17:17:56.245065 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:56.244573 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 22 17:17:57.244608 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:57.244569 2577 patch_prober.go:28] interesting pod/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 22 17:17:57.245594 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:57.245191 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 22 17:17:58.248197 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:58.248169 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:58.248583 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:58.248436 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:17:58.249205 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:17:58.249187 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:18:08.530299 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:18:08.530266 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:18:20.234857 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:18:20.234822 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:18:58.632537 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:18:58.632497 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:19:11.318505 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:11.318426 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:19:17.264530 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:17.264493 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l"] Jul 22 17:19:17.265021 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:17.264752 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" containerID="cri-o://24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b" gracePeriod=30 Jul 22 17:19:21.785420 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:21.785380 2577 patch_prober.go:28] interesting pod/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 17:19:21.785791 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:21.785457 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 17:19:22.506295 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.506271 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:19:22.519414 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.519384 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-istio-data\") pod \"31e78813-8b78-4dc0-a1b6-830856865824\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " Jul 22 17:19:22.519414 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.519416 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-workload-socket\") pod \"31e78813-8b78-4dc0-a1b6-830856865824\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " Jul 22 17:19:22.519604 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.519444 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e78813-8b78-4dc0-a1b6-830856865824-istio-token\") pod \"31e78813-8b78-4dc0-a1b6-830856865824\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " Jul 22 17:19:22.519604 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.519471 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e78813-8b78-4dc0-a1b6-830856865824-istiod-ca-cert\") pod \"31e78813-8b78-4dc0-a1b6-830856865824\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " Jul 22 17:19:22.519604 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.519504 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e78813-8b78-4dc0-a1b6-830856865824-istio-podinfo\") pod \"31e78813-8b78-4dc0-a1b6-830856865824\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " Jul 22 17:19:22.519774 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.519653 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "31e78813-8b78-4dc0-a1b6-830856865824" (UID: "31e78813-8b78-4dc0-a1b6-830856865824"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:19:22.519774 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.519665 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-credential-socket\") pod \"31e78813-8b78-4dc0-a1b6-830856865824\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " Jul 22 17:19:22.519774 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.519703 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djcsg\" (UniqueName: \"kubernetes.io/projected/31e78813-8b78-4dc0-a1b6-830856865824-kube-api-access-djcsg\") pod \"31e78813-8b78-4dc0-a1b6-830856865824\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " Jul 22 17:19:22.519774 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.519768 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-workload-certs\") pod \"31e78813-8b78-4dc0-a1b6-830856865824\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " Jul 22 17:19:22.520023 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.519797 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-istio-envoy\") pod \"31e78813-8b78-4dc0-a1b6-830856865824\" (UID: \"31e78813-8b78-4dc0-a1b6-830856865824\") " Jul 22 17:19:22.520023 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.519865 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e78813-8b78-4dc0-a1b6-830856865824-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "31e78813-8b78-4dc0-a1b6-830856865824" (UID: "31e78813-8b78-4dc0-a1b6-830856865824"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:19:22.520023 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.519982 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "31e78813-8b78-4dc0-a1b6-830856865824" (UID: "31e78813-8b78-4dc0-a1b6-830856865824"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:19:22.520221 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.520181 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "31e78813-8b78-4dc0-a1b6-830856865824" (UID: "31e78813-8b78-4dc0-a1b6-830856865824"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:19:22.520305 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.520230 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-workload-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:22.520305 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.520252 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e78813-8b78-4dc0-a1b6-830856865824-istiod-ca-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:22.520305 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.520266 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-credential-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:22.520305 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.520274 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-istio-data" (OuterVolumeSpecName: "istio-data") pod "31e78813-8b78-4dc0-a1b6-830856865824" (UID: "31e78813-8b78-4dc0-a1b6-830856865824"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:19:22.521714 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.521688 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31e78813-8b78-4dc0-a1b6-830856865824-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "31e78813-8b78-4dc0-a1b6-830856865824" (UID: "31e78813-8b78-4dc0-a1b6-830856865824"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 17:19:22.521848 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.521780 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e78813-8b78-4dc0-a1b6-830856865824-kube-api-access-djcsg" (OuterVolumeSpecName: "kube-api-access-djcsg") pod "31e78813-8b78-4dc0-a1b6-830856865824" (UID: "31e78813-8b78-4dc0-a1b6-830856865824"). InnerVolumeSpecName "kube-api-access-djcsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:19:22.521848 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.521807 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e78813-8b78-4dc0-a1b6-830856865824-istio-token" (OuterVolumeSpecName: "istio-token") pod "31e78813-8b78-4dc0-a1b6-830856865824" (UID: "31e78813-8b78-4dc0-a1b6-830856865824"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:19:22.521928 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.521892 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "31e78813-8b78-4dc0-a1b6-830856865824" (UID: "31e78813-8b78-4dc0-a1b6-830856865824"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:19:22.621311 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.621287 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djcsg\" (UniqueName: \"kubernetes.io/projected/31e78813-8b78-4dc0-a1b6-830856865824-kube-api-access-djcsg\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:22.621311 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.621307 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-workload-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:22.621311 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.621316 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-istio-envoy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:22.621507 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.621323 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e78813-8b78-4dc0-a1b6-830856865824-istio-data\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:22.621507 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.621330 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e78813-8b78-4dc0-a1b6-830856865824-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:22.621507 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:22.621337 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e78813-8b78-4dc0-a1b6-830856865824-istio-podinfo\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:23.290139 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:23.290104 2577 generic.go:356] "Generic (PLEG): container finished" podID="31e78813-8b78-4dc0-a1b6-830856865824" containerID="24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b" exitCode=0 Jul 22 17:19:23.290567 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:23.290142 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" event={"ID":"31e78813-8b78-4dc0-a1b6-830856865824","Type":"ContainerDied","Data":"24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b"} Jul 22 17:19:23.290567 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:23.290187 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" event={"ID":"31e78813-8b78-4dc0-a1b6-830856865824","Type":"ContainerDied","Data":"6ef98f34298f03a1364e137872cf96706015522d637fbee2044645a9921fb8f0"} Jul 22 17:19:23.290567 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:23.290148 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l" Jul 22 17:19:23.290567 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:23.290216 2577 scope.go:117] "RemoveContainer" containerID="24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b" Jul 22 17:19:23.299262 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:23.299245 2577 scope.go:117] "RemoveContainer" containerID="4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a" Jul 22 17:19:23.306703 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:23.306687 2577 scope.go:117] "RemoveContainer" containerID="24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b" Jul 22 17:19:23.306978 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:19:23.306958 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b\": container with ID starting with 24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b not found: ID does not exist" containerID="24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b" Jul 22 17:19:23.307060 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:23.306990 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b"} err="failed to get container status \"24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b\": rpc error: code = NotFound desc = could not find container \"24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b\": container with ID starting with 24adc2a2a581ce655b35d0efa93962fcb281bab66bf1668231c815c012ca477b not found: ID does not exist" Jul 22 17:19:23.307060 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:23.307013 2577 scope.go:117] "RemoveContainer" containerID="4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a" Jul 22 17:19:23.307261 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:19:23.307244 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a\": container with ID starting with 4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a not found: ID does not exist" containerID="4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a" Jul 22 17:19:23.307339 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:23.307268 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a"} err="failed to get container status \"4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a\": rpc error: code = NotFound desc = could not find container \"4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a\": container with ID starting with 4a41d831bcc53f3325f989a7733c7e16b996aa70f11cd9202591936169eec74a not found: ID does not exist" Jul 22 17:19:23.315882 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:23.315862 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l"] Jul 22 17:19:23.320356 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:23.320336 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-668a76-openshift-default-76bd9b6489-7tj7l"] Jul 22 17:19:24.185565 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:24.185534 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31e78813-8b78-4dc0-a1b6-830856865824" path="/var/lib/kubelet/pods/31e78813-8b78-4dc0-a1b6-830856865824/volumes" Jul 22 17:19:31.404295 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:31.404260 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:19:40.683645 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:40.683613 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz"] Jul 22 17:19:40.684132 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:40.683830 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerName="istio-proxy" containerID="cri-o://750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db" gracePeriod=30 Jul 22 17:19:41.759867 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.759827 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj"] Jul 22 17:19:41.760339 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.760209 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" Jul 22 17:19:41.760339 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.760220 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" Jul 22 17:19:41.760339 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.760291 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" Jul 22 17:19:41.760339 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.760300 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" Jul 22 17:19:41.763419 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.763400 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.765839 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:19:41.765810 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" podUID="f30b3400-c090-429b-ad29-aad34cf05909" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.766144 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:19:41.766118 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-api-a-7074f6-openshift-default-dockercfg-ndtss\" is forbidden: User \"system:node:ip-10-0-136-75.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-75.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7074f6-openshift-default-dockercfg-ndtss\"" type="*v1.Secret" Jul 22 17:19:41.798567 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.798533 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj"] Jul 22 17:19:41.877057 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.877022 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-istio-envoy\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.877233 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.877111 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-workload-certs\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.877233 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.877178 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30b3400-c090-429b-ad29-aad34cf05909-istio-podinfo\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.877233 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.877193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30b3400-c090-429b-ad29-aad34cf05909-istiod-ca-cert\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.877338 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.877279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30b3400-c090-429b-ad29-aad34cf05909-istio-token\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.877338 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.877298 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-credential-socket\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.877338 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.877319 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcs9q\" (UniqueName: \"kubernetes.io/projected/f30b3400-c090-429b-ad29-aad34cf05909-kube-api-access-zcs9q\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.877447 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.877391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-workload-socket\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.877447 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.877417 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-istio-data\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.978616 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.978580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-istio-data\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.978616 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.978619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-istio-envoy\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.978903 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.978662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-workload-certs\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.978903 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.978708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30b3400-c090-429b-ad29-aad34cf05909-istio-podinfo\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.978903 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.978883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30b3400-c090-429b-ad29-aad34cf05909-istiod-ca-cert\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.979065 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.978958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30b3400-c090-429b-ad29-aad34cf05909-istio-token\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.979065 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.978986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-credential-socket\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.979065 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.979017 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcs9q\" (UniqueName: \"kubernetes.io/projected/f30b3400-c090-429b-ad29-aad34cf05909-kube-api-access-zcs9q\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.979065 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.979046 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-istio-data\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.979268 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.979076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-workload-certs\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.979268 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.979106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-workload-socket\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.979413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.979299 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-credential-socket\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.979413 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.979362 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-workload-socket\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.979619 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.979592 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30b3400-c090-429b-ad29-aad34cf05909-istiod-ca-cert\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.981135 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.981114 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-istio-envoy\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.981311 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.981291 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30b3400-c090-429b-ad29-aad34cf05909-istio-podinfo\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.990257 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.990232 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcs9q\" (UniqueName: \"kubernetes.io/projected/f30b3400-c090-429b-ad29-aad34cf05909-kube-api-access-zcs9q\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:41.993471 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:41.993448 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30b3400-c090-429b-ad29-aad34cf05909-istio-token\") pod \"e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:42.805001 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:42.804969 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7074f6-openshift-default-dockercfg-ndtss\"" Jul 22 17:19:42.813965 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:42.813948 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:42.972801 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:42.972765 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj"] Jul 22 17:19:42.980126 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:19:42.980104 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30b3400_c090_429b_ad29_aad34cf05909.slice/crio-e8a7a323073e00e1208b5c3a54455be570f3679e251b54909e1939bb3e538d02 WatchSource:0}: Error finding container e8a7a323073e00e1208b5c3a54455be570f3679e251b54909e1939bb3e538d02: Status 404 returned error can't find the container with id e8a7a323073e00e1208b5c3a54455be570f3679e251b54909e1939bb3e538d02 Jul 22 17:19:42.982290 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:42.982259 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:19:42.982369 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:42.982324 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:19:42.982369 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:42.982353 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:19:43.250600 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:43.250566 2577 patch_prober.go:28] interesting pod/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 17:19:43.250788 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:43.250625 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 17:19:43.385253 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:43.385219 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" event={"ID":"f30b3400-c090-429b-ad29-aad34cf05909","Type":"ContainerStarted","Data":"03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7"} Jul 22 17:19:43.385389 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:43.385273 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" event={"ID":"f30b3400-c090-429b-ad29-aad34cf05909","Type":"ContainerStarted","Data":"e8a7a323073e00e1208b5c3a54455be570f3679e251b54909e1939bb3e538d02"} Jul 22 17:19:43.414819 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:43.414780 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" podStartSLOduration=2.414766001 podStartE2EDuration="2.414766001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:19:43.414494668 +0000 UTC m=+2365.770890984" watchObservedRunningTime="2026-07-22 17:19:43.414766001 +0000 UTC m=+2365.771162312" Jul 22 17:19:43.814309 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:43.814281 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:44.819295 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:44.819262 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:45.394496 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:45.394464 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:45.395310 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:45.395290 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:19:45.944827 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:45.944802 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:19:46.013915 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.013847 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-credential-socket\") pod \"02fbc22f-f9c3-482a-ac5e-e13745edd384\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " Jul 22 17:19:46.013915 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.013900 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-workload-certs\") pod \"02fbc22f-f9c3-482a-ac5e-e13745edd384\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " Jul 22 17:19:46.014098 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.013936 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-token\") pod \"02fbc22f-f9c3-482a-ac5e-e13745edd384\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " Jul 22 17:19:46.014098 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.013992 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02fbc22f-f9c3-482a-ac5e-e13745edd384-istiod-ca-cert\") pod \"02fbc22f-f9c3-482a-ac5e-e13745edd384\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " Jul 22 17:19:46.014098 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.014017 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-data\") pod \"02fbc22f-f9c3-482a-ac5e-e13745edd384\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " Jul 22 17:19:46.014098 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.014052 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-workload-socket\") pod \"02fbc22f-f9c3-482a-ac5e-e13745edd384\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " Jul 22 17:19:46.014298 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.014108 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-podinfo\") pod \"02fbc22f-f9c3-482a-ac5e-e13745edd384\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " Jul 22 17:19:46.014298 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.014138 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "02fbc22f-f9c3-482a-ac5e-e13745edd384" (UID: "02fbc22f-f9c3-482a-ac5e-e13745edd384"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:19:46.014298 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.014147 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-envoy\") pod \"02fbc22f-f9c3-482a-ac5e-e13745edd384\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " Jul 22 17:19:46.014298 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.014194 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "02fbc22f-f9c3-482a-ac5e-e13745edd384" (UID: "02fbc22f-f9c3-482a-ac5e-e13745edd384"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:19:46.014298 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.014216 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67gb8\" (UniqueName: \"kubernetes.io/projected/02fbc22f-f9c3-482a-ac5e-e13745edd384-kube-api-access-67gb8\") pod \"02fbc22f-f9c3-482a-ac5e-e13745edd384\" (UID: \"02fbc22f-f9c3-482a-ac5e-e13745edd384\") " Jul 22 17:19:46.014553 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.014376 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "02fbc22f-f9c3-482a-ac5e-e13745edd384" (UID: "02fbc22f-f9c3-482a-ac5e-e13745edd384"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:19:46.014553 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.014501 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-workload-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:46.014553 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.014516 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-workload-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:46.014553 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.014528 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-credential-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:46.015162 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.015078 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-data" (OuterVolumeSpecName: "istio-data") pod "02fbc22f-f9c3-482a-ac5e-e13745edd384" (UID: "02fbc22f-f9c3-482a-ac5e-e13745edd384"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:19:46.015982 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.015387 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02fbc22f-f9c3-482a-ac5e-e13745edd384-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "02fbc22f-f9c3-482a-ac5e-e13745edd384" (UID: "02fbc22f-f9c3-482a-ac5e-e13745edd384"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:19:46.016636 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.016611 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-token" (OuterVolumeSpecName: "istio-token") pod "02fbc22f-f9c3-482a-ac5e-e13745edd384" (UID: "02fbc22f-f9c3-482a-ac5e-e13745edd384"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:19:46.016636 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.016619 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fbc22f-f9c3-482a-ac5e-e13745edd384-kube-api-access-67gb8" (OuterVolumeSpecName: "kube-api-access-67gb8") pod "02fbc22f-f9c3-482a-ac5e-e13745edd384" (UID: "02fbc22f-f9c3-482a-ac5e-e13745edd384"). InnerVolumeSpecName "kube-api-access-67gb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:19:46.017285 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.017263 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "02fbc22f-f9c3-482a-ac5e-e13745edd384" (UID: "02fbc22f-f9c3-482a-ac5e-e13745edd384"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:19:46.017356 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.017313 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "02fbc22f-f9c3-482a-ac5e-e13745edd384" (UID: "02fbc22f-f9c3-482a-ac5e-e13745edd384"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 17:19:46.115890 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.115865 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:46.115890 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.115888 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02fbc22f-f9c3-482a-ac5e-e13745edd384-istiod-ca-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:46.116052 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.115897 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-data\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:46.116052 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.115914 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-podinfo\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:46.116052 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.115923 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02fbc22f-f9c3-482a-ac5e-e13745edd384-istio-envoy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:46.116052 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.115931 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67gb8\" (UniqueName: \"kubernetes.io/projected/02fbc22f-f9c3-482a-ac5e-e13745edd384-kube-api-access-67gb8\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:19:46.401863 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.401833 2577 generic.go:356] "Generic (PLEG): container finished" podID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerID="750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db" exitCode=0 Jul 22 17:19:46.402027 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.401887 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" Jul 22 17:19:46.402027 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.401917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" event={"ID":"02fbc22f-f9c3-482a-ac5e-e13745edd384","Type":"ContainerDied","Data":"750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db"} Jul 22 17:19:46.402027 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.401967 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz" event={"ID":"02fbc22f-f9c3-482a-ac5e-e13745edd384","Type":"ContainerDied","Data":"80e49de7415b36b74f0c3cbeb62ed995e2ec2a488be7d41d8af3ec8be2497171"} Jul 22 17:19:46.402027 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.401989 2577 scope.go:117] "RemoveContainer" containerID="750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db" Jul 22 17:19:46.410372 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.410356 2577 scope.go:117] "RemoveContainer" containerID="96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164" Jul 22 17:19:46.417781 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.417762 2577 scope.go:117] "RemoveContainer" containerID="750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db" Jul 22 17:19:46.418019 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:19:46.418003 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db\": container with ID starting with 750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db not found: ID does not exist" containerID="750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db" Jul 22 17:19:46.418089 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.418028 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db"} err="failed to get container status \"750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db\": rpc error: code = NotFound desc = could not find container \"750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db\": container with ID starting with 750c543d7d0f1f059787b18a2c0983b9d414b314a5aff6f7c43da1e8394392db not found: ID does not exist" Jul 22 17:19:46.418089 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.418043 2577 scope.go:117] "RemoveContainer" containerID="96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164" Jul 22 17:19:46.418286 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:19:46.418269 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164\": container with ID starting with 96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164 not found: ID does not exist" containerID="96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164" Jul 22 17:19:46.418330 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.418292 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164"} err="failed to get container status \"96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164\": rpc error: code = NotFound desc = could not find container \"96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164\": container with ID starting with 96481d66c6e8077631d9a480292f355de42901ea421c28f873cecf34b19fd164 not found: ID does not exist" Jul 22 17:19:46.445263 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.445239 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz"] Jul 22 17:19:46.459004 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:46.458980 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f854fd-openshift-default-868cf5ff45-w8djz"] Jul 22 17:19:48.185293 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:48.185260 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" path="/var/lib/kubelet/pods/02fbc22f-f9c3-482a-ac5e-e13745edd384/volumes" Jul 22 17:19:50.756828 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.756796 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7"] Jul 22 17:19:50.757300 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.757287 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" Jul 22 17:19:50.757349 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.757303 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e78813-8b78-4dc0-a1b6-830856865824" containerName="istio-proxy" Jul 22 17:19:50.757349 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.757332 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerName="istio-proxy" Jul 22 17:19:50.757349 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.757341 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerName="istio-proxy" Jul 22 17:19:50.757495 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.757349 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerName="istio-proxy" Jul 22 17:19:50.757495 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.757358 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerName="istio-proxy" Jul 22 17:19:50.757495 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.757426 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerName="istio-proxy" Jul 22 17:19:50.761913 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.761895 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.764270 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.764244 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-050beb-openshift-default-dockercfg-mlnwq\"" Jul 22 17:19:50.781174 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.781143 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7"] Jul 22 17:19:50.857921 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.857891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-podinfo\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.858129 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.857939 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-data\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.858129 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.857963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-token\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.858129 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.858010 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-envoy\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.858129 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.858040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-workload-socket\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.858129 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.858071 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65838fa5-6afa-4c90-84e3-23d8478024f8-istiod-ca-cert\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.858129 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.858100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-workload-certs\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.858423 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.858198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9x7\" (UniqueName: \"kubernetes.io/projected/65838fa5-6afa-4c90-84e3-23d8478024f8-kube-api-access-zn9x7\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.858423 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.858248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-credential-socket\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959173 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-podinfo\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959333 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-data\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959333 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-token\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959333 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-envoy\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959333 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959273 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-workload-socket\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959333 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65838fa5-6afa-4c90-84e3-23d8478024f8-istiod-ca-cert\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959333 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-workload-certs\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959658 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9x7\" (UniqueName: \"kubernetes.io/projected/65838fa5-6afa-4c90-84e3-23d8478024f8-kube-api-access-zn9x7\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959658 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-credential-socket\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959658 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-data\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959839 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959699 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-workload-certs\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959839 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959800 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-workload-socket\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.959839 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.959832 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-credential-socket\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.960252 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.960231 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65838fa5-6afa-4c90-84e3-23d8478024f8-istiod-ca-cert\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.961831 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.961809 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-envoy\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.962041 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.962025 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-podinfo\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.969670 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.969648 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-token\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:50.970136 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:50.970119 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9x7\" (UniqueName: \"kubernetes.io/projected/65838fa5-6afa-4c90-84e3-23d8478024f8-kube-api-access-zn9x7\") pod \"e2e-api-b-050beb-openshift-default-69d464889-ngwt7\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:51.071867 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:51.071786 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:51.203614 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:51.203584 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:19:51.216770 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:51.216730 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7"] Jul 22 17:19:51.218645 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:19:51.218618 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65838fa5_6afa_4c90_84e3_23d8478024f8.slice/crio-7b95bd974aa93ad9f6c16892c813e5483acf33316e65e1be1034fd98d2d81c5d WatchSource:0}: Error finding container 7b95bd974aa93ad9f6c16892c813e5483acf33316e65e1be1034fd98d2d81c5d: Status 404 returned error can't find the container with id 7b95bd974aa93ad9f6c16892c813e5483acf33316e65e1be1034fd98d2d81c5d Jul 22 17:19:51.220904 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:51.220873 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:19:51.220991 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:51.220941 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:19:51.220991 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:51.220982 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:19:51.427507 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:51.427469 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" event={"ID":"65838fa5-6afa-4c90-84e3-23d8478024f8","Type":"ContainerStarted","Data":"1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a"} Jul 22 17:19:51.427700 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:51.427534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" event={"ID":"65838fa5-6afa-4c90-84e3-23d8478024f8","Type":"ContainerStarted","Data":"7b95bd974aa93ad9f6c16892c813e5483acf33316e65e1be1034fd98d2d81c5d"} Jul 22 17:19:51.453661 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:51.453607 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" podStartSLOduration=1.453591675 podStartE2EDuration="1.453591675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:19:51.453143783 +0000 UTC m=+2373.809540102" watchObservedRunningTime="2026-07-22 17:19:51.453591675 +0000 UTC m=+2373.809987992" Jul 22 17:19:52.072728 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:52.072697 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:52.077341 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:52.077316 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:52.432710 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:52.432678 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:52.433684 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:52.433660 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:19:59.473372 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:59.473340 2577 generic.go:356] "Generic (PLEG): container finished" podID="f30b3400-c090-429b-ad29-aad34cf05909" containerID="03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7" exitCode=0 Jul 22 17:19:59.473776 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:59.473409 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" event={"ID":"f30b3400-c090-429b-ad29-aad34cf05909","Type":"ContainerDied","Data":"03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7"} Jul 22 17:19:59.473819 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:59.473801 2577 scope.go:117] "RemoveContainer" containerID="03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7" Jul 22 17:19:59.474962 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:59.474936 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:19:59.475019 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:59.475011 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:19:59.475054 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:19:59.475041 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:20:00.395542 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:00.395495 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:20:00.481849 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:00.481806 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" event={"ID":"f30b3400-c090-429b-ad29-aad34cf05909","Type":"ContainerStarted","Data":"369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b"} Jul 22 17:20:00.814593 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:00.814552 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:20:00.815579 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:00.815555 2577 patch_prober.go:28] interesting pod/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 17:20:00.815682 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:00.815623 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" podUID="f30b3400-c090-429b-ad29-aad34cf05909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 17:20:01.814682 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:01.814642 2577 patch_prober.go:28] interesting pod/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 17:20:01.815186 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:01.814695 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" podUID="f30b3400-c090-429b-ad29-aad34cf05909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 17:20:02.815153 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:02.815115 2577 patch_prober.go:28] interesting pod/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 17:20:02.815566 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:02.815181 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" podUID="f30b3400-c090-429b-ad29-aad34cf05909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 17:20:03.815031 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:03.814982 2577 patch_prober.go:28] interesting pod/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 17:20:03.815207 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:03.815079 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" podUID="f30b3400-c090-429b-ad29-aad34cf05909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 17:20:04.819029 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:04.818998 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:20:04.819509 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:04.819269 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:20:04.820055 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:04.820033 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:20:05.010486 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:05.010457 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:20:07.434860 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:07.434828 2577 patch_prober.go:28] interesting pod/e2e-api-b-050beb-openshift-default-69d464889-ngwt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 22 17:20:07.435265 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:07.434883 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 22 17:20:07.521653 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:07.521620 2577 generic.go:356] "Generic (PLEG): container finished" podID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerID="1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a" exitCode=0 Jul 22 17:20:07.521807 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:07.521670 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" event={"ID":"65838fa5-6afa-4c90-84e3-23d8478024f8","Type":"ContainerDied","Data":"1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a"} Jul 22 17:20:07.522074 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:07.522060 2577 scope.go:117] "RemoveContainer" containerID="1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a" Jul 22 17:20:07.523002 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:07.522976 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:20:07.523058 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:07.523042 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:20:07.523094 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:07.523077 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:20:08.527979 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:08.527927 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" event={"ID":"65838fa5-6afa-4c90-84e3-23d8478024f8","Type":"ContainerStarted","Data":"fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d"} Jul 22 17:20:09.072907 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:09.072873 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:20:09.073105 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:09.073081 2577 patch_prober.go:28] interesting pod/e2e-api-b-050beb-openshift-default-69d464889-ngwt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 22 17:20:09.073150 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:09.073132 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 22 17:20:10.072353 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:10.072310 2577 patch_prober.go:28] interesting pod/e2e-api-b-050beb-openshift-default-69d464889-ngwt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 22 17:20:10.072765 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:10.072373 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 22 17:20:11.072827 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:11.072787 2577 patch_prober.go:28] interesting pod/e2e-api-b-050beb-openshift-default-69d464889-ngwt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 22 17:20:11.073213 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:11.072839 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 22 17:20:12.072532 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:12.072493 2577 patch_prober.go:28] interesting pod/e2e-api-b-050beb-openshift-default-69d464889-ngwt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 22 17:20:12.072767 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:12.072558 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 22 17:20:13.076712 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:13.076684 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:20:13.077209 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:13.077089 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:20:13.077604 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:13.077585 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:20:24.893348 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:24.893313 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:20:33.719185 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:33.719153 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj"] Jul 22 17:20:33.719717 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:33.719388 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" podUID="f30b3400-c090-429b-ad29-aad34cf05909" containerName="istio-proxy" containerID="cri-o://369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b" gracePeriod=30 Jul 22 17:20:34.821837 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:34.821805 2577 patch_prober.go:28] interesting pod/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 17:20:34.822208 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:34.821861 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" podUID="f30b3400-c090-429b-ad29-aad34cf05909" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 17:20:38.967174 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:38.967151 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:20:39.097031 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.096947 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30b3400-c090-429b-ad29-aad34cf05909-istio-token\") pod \"f30b3400-c090-429b-ad29-aad34cf05909\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " Jul 22 17:20:39.097031 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097001 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-workload-certs\") pod \"f30b3400-c090-429b-ad29-aad34cf05909\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " Jul 22 17:20:39.097031 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097029 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-workload-socket\") pod \"f30b3400-c090-429b-ad29-aad34cf05909\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " Jul 22 17:20:39.097312 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097055 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30b3400-c090-429b-ad29-aad34cf05909-istio-podinfo\") pod \"f30b3400-c090-429b-ad29-aad34cf05909\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " Jul 22 17:20:39.097312 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097082 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30b3400-c090-429b-ad29-aad34cf05909-istiod-ca-cert\") pod \"f30b3400-c090-429b-ad29-aad34cf05909\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " Jul 22 17:20:39.097312 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097107 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-credential-socket\") pod \"f30b3400-c090-429b-ad29-aad34cf05909\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " Jul 22 17:20:39.097312 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097150 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcs9q\" (UniqueName: \"kubernetes.io/projected/f30b3400-c090-429b-ad29-aad34cf05909-kube-api-access-zcs9q\") pod \"f30b3400-c090-429b-ad29-aad34cf05909\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " Jul 22 17:20:39.097312 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097190 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-istio-envoy\") pod \"f30b3400-c090-429b-ad29-aad34cf05909\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " Jul 22 17:20:39.097312 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097280 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-istio-data\") pod \"f30b3400-c090-429b-ad29-aad34cf05909\" (UID: \"f30b3400-c090-429b-ad29-aad34cf05909\") " Jul 22 17:20:39.097573 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097390 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f30b3400-c090-429b-ad29-aad34cf05909" (UID: "f30b3400-c090-429b-ad29-aad34cf05909"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:20:39.097573 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097413 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f30b3400-c090-429b-ad29-aad34cf05909" (UID: "f30b3400-c090-429b-ad29-aad34cf05909"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:20:39.097573 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097426 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f30b3400-c090-429b-ad29-aad34cf05909" (UID: "f30b3400-c090-429b-ad29-aad34cf05909"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:20:39.097686 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097641 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-workload-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:20:39.097686 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097669 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-workload-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:20:39.097815 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097685 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-credential-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:20:39.097867 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.097825 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30b3400-c090-429b-ad29-aad34cf05909-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f30b3400-c090-429b-ad29-aad34cf05909" (UID: "f30b3400-c090-429b-ad29-aad34cf05909"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:20:39.098162 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.098141 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-istio-data" (OuterVolumeSpecName: "istio-data") pod "f30b3400-c090-429b-ad29-aad34cf05909" (UID: "f30b3400-c090-429b-ad29-aad34cf05909"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:20:39.099345 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.099324 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30b3400-c090-429b-ad29-aad34cf05909-kube-api-access-zcs9q" (OuterVolumeSpecName: "kube-api-access-zcs9q") pod "f30b3400-c090-429b-ad29-aad34cf05909" (UID: "f30b3400-c090-429b-ad29-aad34cf05909"). InnerVolumeSpecName "kube-api-access-zcs9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:20:39.099460 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.099428 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f30b3400-c090-429b-ad29-aad34cf05909" (UID: "f30b3400-c090-429b-ad29-aad34cf05909"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:20:39.099550 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.099532 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f30b3400-c090-429b-ad29-aad34cf05909-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f30b3400-c090-429b-ad29-aad34cf05909" (UID: "f30b3400-c090-429b-ad29-aad34cf05909"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 17:20:39.099588 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.099549 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30b3400-c090-429b-ad29-aad34cf05909-istio-token" (OuterVolumeSpecName: "istio-token") pod "f30b3400-c090-429b-ad29-aad34cf05909" (UID: "f30b3400-c090-429b-ad29-aad34cf05909"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:20:39.198815 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.198779 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-istio-data\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:20:39.198815 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.198812 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30b3400-c090-429b-ad29-aad34cf05909-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:20:39.198815 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.198821 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30b3400-c090-429b-ad29-aad34cf05909-istio-podinfo\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:20:39.198815 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.198829 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30b3400-c090-429b-ad29-aad34cf05909-istiod-ca-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:20:39.199057 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.198837 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcs9q\" (UniqueName: \"kubernetes.io/projected/f30b3400-c090-429b-ad29-aad34cf05909-kube-api-access-zcs9q\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:20:39.199057 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.198847 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30b3400-c090-429b-ad29-aad34cf05909-istio-envoy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:20:39.669556 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.669523 2577 generic.go:356] "Generic (PLEG): container finished" podID="f30b3400-c090-429b-ad29-aad34cf05909" containerID="369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b" exitCode=0 Jul 22 17:20:39.669717 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.669567 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" event={"ID":"f30b3400-c090-429b-ad29-aad34cf05909","Type":"ContainerDied","Data":"369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b"} Jul 22 17:20:39.669717 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.669609 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" event={"ID":"f30b3400-c090-429b-ad29-aad34cf05909","Type":"ContainerDied","Data":"e8a7a323073e00e1208b5c3a54455be570f3679e251b54909e1939bb3e538d02"} Jul 22 17:20:39.669717 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.669629 2577 scope.go:117] "RemoveContainer" containerID="369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b" Jul 22 17:20:39.669717 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.669567 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj" Jul 22 17:20:39.678539 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.678524 2577 scope.go:117] "RemoveContainer" containerID="03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7" Jul 22 17:20:39.686061 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.686045 2577 scope.go:117] "RemoveContainer" containerID="369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b" Jul 22 17:20:39.686303 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:20:39.686284 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b\": container with ID starting with 369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b not found: ID does not exist" containerID="369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b" Jul 22 17:20:39.686373 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.686314 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b"} err="failed to get container status \"369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b\": rpc error: code = NotFound desc = could not find container \"369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b\": container with ID starting with 369ee7474058a99a16a7e76c50a7ece00536a01913673c5b877e0f890f40f79b not found: ID does not exist" Jul 22 17:20:39.686373 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.686338 2577 scope.go:117] "RemoveContainer" containerID="03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7" Jul 22 17:20:39.686590 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:20:39.686573 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7\": container with ID starting with 03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7 not found: ID does not exist" containerID="03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7" Jul 22 17:20:39.686625 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.686595 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7"} err="failed to get container status \"03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7\": rpc error: code = NotFound desc = could not find container \"03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7\": container with ID starting with 03e87fbffe1435552d7f6798a8042da5ded024f4a8b86671bbb972eac740dbb7 not found: ID does not exist" Jul 22 17:20:39.690211 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.690190 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj"] Jul 22 17:20:39.694226 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:39.694207 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7074f6-openshift-default-5d579bcf75-xhgjj"] Jul 22 17:20:40.185204 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:40.185165 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f30b3400-c090-429b-ad29-aad34cf05909" path="/var/lib/kubelet/pods/f30b3400-c090-429b-ad29-aad34cf05909/volumes" Jul 22 17:20:54.892985 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:20:54.892945 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:21:01.042163 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.042125 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7"] Jul 22 17:21:01.042584 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.042402 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerName="istio-proxy" containerID="cri-o://fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d" gracePeriod=30 Jul 22 17:21:01.968381 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.968344 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5"] Jul 22 17:21:01.968705 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.968693 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30b3400-c090-429b-ad29-aad34cf05909" containerName="istio-proxy" Jul 22 17:21:01.968774 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.968707 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30b3400-c090-429b-ad29-aad34cf05909" containerName="istio-proxy" Jul 22 17:21:01.968774 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.968716 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30b3400-c090-429b-ad29-aad34cf05909" containerName="istio-proxy" Jul 22 17:21:01.968774 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.968722 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30b3400-c090-429b-ad29-aad34cf05909" containerName="istio-proxy" Jul 22 17:21:01.968873 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.968799 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="02fbc22f-f9c3-482a-ac5e-e13745edd384" containerName="istio-proxy" Jul 22 17:21:01.968873 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.968808 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30b3400-c090-429b-ad29-aad34cf05909" containerName="istio-proxy" Jul 22 17:21:01.968873 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.968816 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30b3400-c090-429b-ad29-aad34cf05909" containerName="istio-proxy" Jul 22 17:21:01.970784 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.970763 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:01.973072 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.973052 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-629c13-openshift-default-dockercfg-mpbf7\"" Jul 22 17:21:01.985394 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:01.985366 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5"] Jul 22 17:21:02.094287 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.094257 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-workload-socket\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.094669 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.094301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-workload-certs\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.094669 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.094325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ceb759-f283-4f61-90d6-2ba297658818-istio-podinfo\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.094669 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.094409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-istio-envoy\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.094669 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.094438 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ceb759-f283-4f61-90d6-2ba297658818-istio-token\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.094669 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.094474 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m2wj\" (UniqueName: \"kubernetes.io/projected/28ceb759-f283-4f61-90d6-2ba297658818-kube-api-access-9m2wj\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.094669 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.094523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ceb759-f283-4f61-90d6-2ba297658818-istiod-ca-cert\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.094669 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.094569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-credential-socket\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.094669 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.094595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-istio-data\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.195753 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.195707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-credential-socket\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.195940 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.195771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-istio-data\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.195940 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.195811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-workload-socket\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.195940 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.195856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-workload-certs\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.195940 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.195886 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ceb759-f283-4f61-90d6-2ba297658818-istio-podinfo\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.195940 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.195930 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-istio-envoy\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.196202 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.195954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ceb759-f283-4f61-90d6-2ba297658818-istio-token\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.196202 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.195999 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m2wj\" (UniqueName: \"kubernetes.io/projected/28ceb759-f283-4f61-90d6-2ba297658818-kube-api-access-9m2wj\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.196202 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.196041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ceb759-f283-4f61-90d6-2ba297658818-istiod-ca-cert\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.196202 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.196181 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-credential-socket\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.196395 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.196209 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-workload-socket\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.196581 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.196559 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-istio-data\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.196681 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.196661 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-workload-certs\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.196998 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.196982 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ceb759-f283-4f61-90d6-2ba297658818-istiod-ca-cert\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.198884 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.198858 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-istio-envoy\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.199002 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.198983 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ceb759-f283-4f61-90d6-2ba297658818-istio-podinfo\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.204827 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.204807 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ceb759-f283-4f61-90d6-2ba297658818-istio-token\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.204976 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.204956 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m2wj\" (UniqueName: \"kubernetes.io/projected/28ceb759-f283-4f61-90d6-2ba297658818-kube-api-access-9m2wj\") pod \"e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.282296 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.282206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:02.427505 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.427483 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5"] Jul 22 17:21:02.429646 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:21:02.429613 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ceb759_f283_4f61_90d6_2ba297658818.slice/crio-542520be7993e69ba202de5f14b3a457c440e750a5c45243ef646797c5b5b50b WatchSource:0}: Error finding container 542520be7993e69ba202de5f14b3a457c440e750a5c45243ef646797c5b5b50b: Status 404 returned error can't find the container with id 542520be7993e69ba202de5f14b3a457c440e750a5c45243ef646797c5b5b50b Jul 22 17:21:02.431642 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.431612 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:21:02.431754 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.431674 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:21:02.431754 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.431701 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:21:02.779093 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.779058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" event={"ID":"28ceb759-f283-4f61-90d6-2ba297658818","Type":"ContainerStarted","Data":"4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44"} Jul 22 17:21:02.779249 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.779120 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" event={"ID":"28ceb759-f283-4f61-90d6-2ba297658818","Type":"ContainerStarted","Data":"542520be7993e69ba202de5f14b3a457c440e750a5c45243ef646797c5b5b50b"} Jul 22 17:21:02.804021 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:02.803966 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" podStartSLOduration=1.803948417 podStartE2EDuration="1.803948417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:21:02.80236689 +0000 UTC m=+2445.158763224" watchObservedRunningTime="2026-07-22 17:21:02.803948417 +0000 UTC m=+2445.160344735" Jul 22 17:21:03.282908 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:03.282869 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:03.287907 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:03.287875 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:03.784687 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:03.784648 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:03.785479 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:03.785457 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:06.298446 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.298422 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:21:06.432818 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.432789 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-podinfo\") pod \"65838fa5-6afa-4c90-84e3-23d8478024f8\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " Jul 22 17:21:06.433023 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.432830 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-workload-certs\") pod \"65838fa5-6afa-4c90-84e3-23d8478024f8\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " Jul 22 17:21:06.433023 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.432876 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-envoy\") pod \"65838fa5-6afa-4c90-84e3-23d8478024f8\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " Jul 22 17:21:06.433023 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.432897 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-workload-socket\") pod \"65838fa5-6afa-4c90-84e3-23d8478024f8\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " Jul 22 17:21:06.433023 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.432924 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn9x7\" (UniqueName: \"kubernetes.io/projected/65838fa5-6afa-4c90-84e3-23d8478024f8-kube-api-access-zn9x7\") pod \"65838fa5-6afa-4c90-84e3-23d8478024f8\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " Jul 22 17:21:06.433023 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.432945 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-token\") pod \"65838fa5-6afa-4c90-84e3-23d8478024f8\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " Jul 22 17:21:06.433023 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.432985 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-data\") pod \"65838fa5-6afa-4c90-84e3-23d8478024f8\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " Jul 22 17:21:06.433356 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.433058 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65838fa5-6afa-4c90-84e3-23d8478024f8-istiod-ca-cert\") pod \"65838fa5-6afa-4c90-84e3-23d8478024f8\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " Jul 22 17:21:06.433356 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.433082 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-credential-socket\") pod \"65838fa5-6afa-4c90-84e3-23d8478024f8\" (UID: \"65838fa5-6afa-4c90-84e3-23d8478024f8\") " Jul 22 17:21:06.433356 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.433324 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65838fa5-6afa-4c90-84e3-23d8478024f8" (UID: "65838fa5-6afa-4c90-84e3-23d8478024f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:21:06.433597 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.433574 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65838fa5-6afa-4c90-84e3-23d8478024f8" (UID: "65838fa5-6afa-4c90-84e3-23d8478024f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:21:06.433654 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.433631 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65838fa5-6afa-4c90-84e3-23d8478024f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65838fa5-6afa-4c90-84e3-23d8478024f8" (UID: "65838fa5-6afa-4c90-84e3-23d8478024f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:21:06.433654 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.433635 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65838fa5-6afa-4c90-84e3-23d8478024f8" (UID: "65838fa5-6afa-4c90-84e3-23d8478024f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:21:06.433920 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.433898 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "65838fa5-6afa-4c90-84e3-23d8478024f8" (UID: "65838fa5-6afa-4c90-84e3-23d8478024f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:21:06.435254 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.435215 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65838fa5-6afa-4c90-84e3-23d8478024f8" (UID: "65838fa5-6afa-4c90-84e3-23d8478024f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 17:21:06.435254 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.435226 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65838fa5-6afa-4c90-84e3-23d8478024f8-kube-api-access-zn9x7" (OuterVolumeSpecName: "kube-api-access-zn9x7") pod "65838fa5-6afa-4c90-84e3-23d8478024f8" (UID: "65838fa5-6afa-4c90-84e3-23d8478024f8"). InnerVolumeSpecName "kube-api-access-zn9x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:21:06.435503 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.435479 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "65838fa5-6afa-4c90-84e3-23d8478024f8" (UID: "65838fa5-6afa-4c90-84e3-23d8478024f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:21:06.436024 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.436003 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65838fa5-6afa-4c90-84e3-23d8478024f8" (UID: "65838fa5-6afa-4c90-84e3-23d8478024f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:21:06.533895 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.533860 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65838fa5-6afa-4c90-84e3-23d8478024f8-istiod-ca-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:21:06.533895 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.533889 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-credential-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:21:06.534087 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.533902 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-podinfo\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:21:06.534087 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.533916 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-workload-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:21:06.534087 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.533927 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-envoy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:21:06.534087 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.533938 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-workload-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:21:06.534087 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.533950 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn9x7\" (UniqueName: \"kubernetes.io/projected/65838fa5-6afa-4c90-84e3-23d8478024f8-kube-api-access-zn9x7\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:21:06.534087 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.533962 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:21:06.534087 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.533974 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65838fa5-6afa-4c90-84e3-23d8478024f8-istio-data\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:21:06.799778 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.799662 2577 generic.go:356] "Generic (PLEG): container finished" podID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerID="fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d" exitCode=0 Jul 22 17:21:06.799778 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.799710 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" Jul 22 17:21:06.799778 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.799765 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" event={"ID":"65838fa5-6afa-4c90-84e3-23d8478024f8","Type":"ContainerDied","Data":"fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d"} Jul 22 17:21:06.800064 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.799827 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7" event={"ID":"65838fa5-6afa-4c90-84e3-23d8478024f8","Type":"ContainerDied","Data":"7b95bd974aa93ad9f6c16892c813e5483acf33316e65e1be1034fd98d2d81c5d"} Jul 22 17:21:06.800064 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.799858 2577 scope.go:117] "RemoveContainer" containerID="fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d" Jul 22 17:21:06.808922 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.808907 2577 scope.go:117] "RemoveContainer" containerID="1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a" Jul 22 17:21:06.816466 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.816451 2577 scope.go:117] "RemoveContainer" containerID="fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d" Jul 22 17:21:06.816682 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:21:06.816667 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d\": container with ID starting with fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d not found: ID does not exist" containerID="fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d" Jul 22 17:21:06.816749 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.816690 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d"} err="failed to get container status \"fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d\": rpc error: code = NotFound desc = could not find container \"fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d\": container with ID starting with fce00441c2894aee923f54cef47177ef4672a93437f7cb28cf5ed2d66228fb5d not found: ID does not exist" Jul 22 17:21:06.816749 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.816710 2577 scope.go:117] "RemoveContainer" containerID="1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a" Jul 22 17:21:06.816978 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:21:06.816960 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a\": container with ID starting with 1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a not found: ID does not exist" containerID="1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a" Jul 22 17:21:06.817030 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.816984 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a"} err="failed to get container status \"1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a\": rpc error: code = NotFound desc = could not find container \"1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a\": container with ID starting with 1198b62537ef28332d24907e8db50412ae83f93a7fe8384d0a80e21ba6f6af7a not found: ID does not exist" Jul 22 17:21:06.824033 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.824011 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7"] Jul 22 17:21:06.830491 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:06.830469 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-050beb-openshift-default-69d464889-ngwt7"] Jul 22 17:21:08.186114 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:08.186083 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" path="/var/lib/kubelet/pods/65838fa5-6afa-4c90-84e3-23d8478024f8/volumes" Jul 22 17:21:09.912708 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:09.912673 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk"] Jul 22 17:21:09.913194 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:09.913067 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerName="istio-proxy" Jul 22 17:21:09.913194 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:09.913079 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerName="istio-proxy" Jul 22 17:21:09.913194 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:09.913145 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerName="istio-proxy" Jul 22 17:21:09.916057 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:09.916035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:09.919460 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:09.919437 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-05e61d-openshift-default-dockercfg-5wxf9\"" Jul 22 17:21:09.932924 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:09.932902 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk"] Jul 22 17:21:10.063854 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.063821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1413a620-eaf5-4d20-9263-279b44821241-istio-podinfo\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.064010 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.063861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-istio-envoy\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.064010 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.063882 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1413a620-eaf5-4d20-9263-279b44821241-istio-token\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.064010 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.063970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-istio-data\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.064134 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.064012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-credential-socket\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.064134 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.064041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcv9z\" (UniqueName: \"kubernetes.io/projected/1413a620-eaf5-4d20-9263-279b44821241-kube-api-access-tcv9z\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.064134 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.064069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-workload-certs\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.064226 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.064156 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-workload-socket\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.064226 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.064182 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1413a620-eaf5-4d20-9263-279b44821241-istiod-ca-cert\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165179 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-workload-socket\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165179 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1413a620-eaf5-4d20-9263-279b44821241-istiod-ca-cert\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165179 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1413a620-eaf5-4d20-9263-279b44821241-istio-podinfo\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165463 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-istio-envoy\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165463 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1413a620-eaf5-4d20-9263-279b44821241-istio-token\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165463 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-istio-data\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165463 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-credential-socket\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165463 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165368 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcv9z\" (UniqueName: \"kubernetes.io/projected/1413a620-eaf5-4d20-9263-279b44821241-kube-api-access-tcv9z\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165463 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165436 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-workload-certs\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165828 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165508 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-workload-socket\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165828 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165640 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-credential-socket\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165828 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165815 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-istio-data\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165989 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165836 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-workload-certs\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.165989 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.165940 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1413a620-eaf5-4d20-9263-279b44821241-istiod-ca-cert\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.167629 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.167605 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-istio-envoy\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.168353 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.168317 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1413a620-eaf5-4d20-9263-279b44821241-istio-podinfo\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.173570 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.173541 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcv9z\" (UniqueName: \"kubernetes.io/projected/1413a620-eaf5-4d20-9263-279b44821241-kube-api-access-tcv9z\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.173678 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.173653 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1413a620-eaf5-4d20-9263-279b44821241-istio-token\") pod \"e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.226365 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.226331 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:10.358871 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.358832 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk"] Jul 22 17:21:10.361894 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:21:10.361860 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1413a620_eaf5_4d20_9263_279b44821241.slice/crio-220b66b48b9b610d02ae6112d3c41c846972b81ec83cc01e4b8b819062cdd158 WatchSource:0}: Error finding container 220b66b48b9b610d02ae6112d3c41c846972b81ec83cc01e4b8b819062cdd158: Status 404 returned error can't find the container with id 220b66b48b9b610d02ae6112d3c41c846972b81ec83cc01e4b8b819062cdd158 Jul 22 17:21:10.364074 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.364033 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:21:10.364138 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.364120 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:21:10.364183 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.364159 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:21:10.821066 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.821029 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" event={"ID":"1413a620-eaf5-4d20-9263-279b44821241","Type":"ContainerStarted","Data":"fb9d9b18d4ce9e50119d8e387aff11036c6c58c14f1feffeeaf0f5b721cfcb05"} Jul 22 17:21:10.821216 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.821085 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" event={"ID":"1413a620-eaf5-4d20-9263-279b44821241","Type":"ContainerStarted","Data":"220b66b48b9b610d02ae6112d3c41c846972b81ec83cc01e4b8b819062cdd158"} Jul 22 17:21:10.857174 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:10.857121 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" podStartSLOduration=1.857104522 podStartE2EDuration="1.857104522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:21:10.856095291 +0000 UTC m=+2453.212491621" watchObservedRunningTime="2026-07-22 17:21:10.857104522 +0000 UTC m=+2453.213500840" Jul 22 17:21:11.227300 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:11.227261 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:11.232131 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:11.232109 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:11.826232 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:11.826205 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:11.827077 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:11.827059 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:12.717487 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:12.717451 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:21:19.825926 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:19.825881 2577 patch_prober.go:28] interesting pod/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 17:21:19.826308 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:19.825961 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" podUID="28ceb759-f283-4f61-90d6-2ba297658818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 17:21:20.894174 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:20.894140 2577 generic.go:356] "Generic (PLEG): container finished" podID="28ceb759-f283-4f61-90d6-2ba297658818" containerID="4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44" exitCode=0 Jul 22 17:21:20.894663 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:20.894223 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" event={"ID":"28ceb759-f283-4f61-90d6-2ba297658818","Type":"ContainerDied","Data":"4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44"} Jul 22 17:21:20.894663 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:20.894606 2577 scope.go:117] "RemoveContainer" containerID="4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44" Jul 22 17:21:20.895606 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:20.895582 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:21:20.895665 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:20.895648 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:21:20.895703 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:20.895679 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:21:21.903525 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:21.903493 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" event={"ID":"28ceb759-f283-4f61-90d6-2ba297658818","Type":"ContainerStarted","Data":"30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a"} Jul 22 17:21:22.283363 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:22.283266 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:22.283860 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:22.283833 2577 patch_prober.go:28] interesting pod/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 22 17:21:22.284066 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:22.284042 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" podUID="28ceb759-f283-4f61-90d6-2ba297658818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 22 17:21:23.283421 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:23.282954 2577 patch_prober.go:28] interesting pod/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 22 17:21:23.283421 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:23.283020 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" podUID="28ceb759-f283-4f61-90d6-2ba297658818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 22 17:21:24.283285 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:24.283236 2577 patch_prober.go:28] interesting pod/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 22 17:21:24.283507 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:24.283308 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" podUID="28ceb759-f283-4f61-90d6-2ba297658818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 22 17:21:25.283566 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:25.283522 2577 patch_prober.go:28] interesting pod/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 22 17:21:25.284280 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:25.284242 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" podUID="28ceb759-f283-4f61-90d6-2ba297658818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 22 17:21:26.286312 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:26.286285 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:26.286691 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:26.286564 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:26.287426 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:26.287403 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:21:26.802625 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:26.802588 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:21:27.867883 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:27.867848 2577 patch_prober.go:28] interesting pod/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 17:21:27.868374 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:27.867925 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" podUID="1413a620-eaf5-4d20-9263-279b44821241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 17:21:29.940346 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:29.940310 2577 generic.go:356] "Generic (PLEG): container finished" podID="1413a620-eaf5-4d20-9263-279b44821241" containerID="fb9d9b18d4ce9e50119d8e387aff11036c6c58c14f1feffeeaf0f5b721cfcb05" exitCode=0 Jul 22 17:21:29.940789 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:29.940382 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" event={"ID":"1413a620-eaf5-4d20-9263-279b44821241","Type":"ContainerDied","Data":"fb9d9b18d4ce9e50119d8e387aff11036c6c58c14f1feffeeaf0f5b721cfcb05"} Jul 22 17:21:29.940851 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:29.940800 2577 scope.go:117] "RemoveContainer" containerID="fb9d9b18d4ce9e50119d8e387aff11036c6c58c14f1feffeeaf0f5b721cfcb05" Jul 22 17:21:29.941864 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:29.941837 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:21:29.941983 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:29.941919 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:21:29.941983 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:29.941967 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 17:21:30.950077 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:30.950038 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" event={"ID":"1413a620-eaf5-4d20-9263-279b44821241","Type":"ContainerStarted","Data":"50fb73641dc8f86d5903e2935974b553066ecf6d43c5ee3ffc9c42943e537bba"} Jul 22 17:21:31.227259 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:31.227163 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:31.227444 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:31.227419 2577 patch_prober.go:28] interesting pod/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 22 17:21:31.227517 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:31.227482 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" podUID="1413a620-eaf5-4d20-9263-279b44821241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 22 17:21:32.227103 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:32.227067 2577 patch_prober.go:28] interesting pod/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 22 17:21:32.227475 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:32.227120 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" podUID="1413a620-eaf5-4d20-9263-279b44821241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 22 17:21:33.227220 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:33.227184 2577 patch_prober.go:28] interesting pod/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 22 17:21:33.227583 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:33.227244 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" podUID="1413a620-eaf5-4d20-9263-279b44821241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 22 17:21:34.227049 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:34.227014 2577 patch_prober.go:28] interesting pod/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 22 17:21:34.227247 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:34.227069 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" podUID="1413a620-eaf5-4d20-9263-279b44821241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 22 17:21:35.230138 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:35.230109 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:35.230523 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:35.230504 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:35.231238 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:35.231218 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:21:37.197731 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:21:37.197696 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:22:10.001699 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.001666 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl"] Jul 22 17:22:10.002169 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.002046 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerName="istio-proxy" Jul 22 17:22:10.002169 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.002058 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerName="istio-proxy" Jul 22 17:22:10.002169 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.002118 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="65838fa5-6afa-4c90-84e3-23d8478024f8" containerName="istio-proxy" Jul 22 17:22:10.003943 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.003927 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.007297 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.007272 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-05e61d\"/\"openshift-service-ca.crt\"" Jul 22 17:22:10.007427 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.007308 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-05e61d\"/\"auth-test-model-05e61d-kserve-self-signed-certs\"" Jul 22 17:22:10.007427 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.007360 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-05e61d\"/\"kube-root-ca.crt\"" Jul 22 17:22:10.007427 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.007315 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-05e61d\"/\"default-dockercfg-tw7rz\"" Jul 22 17:22:10.014144 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.014123 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl"] Jul 22 17:22:10.174828 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.174796 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-tmp-dir\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.174983 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.174840 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-tls-certs\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.174983 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.174872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-home\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.174983 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.174889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-model-cache\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.174983 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.174904 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgx5j\" (UniqueName: \"kubernetes.io/projected/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-kube-api-access-qgx5j\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.174983 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.174957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-dshm\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.275656 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.275569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-tmp-dir\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.275656 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.275617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-tls-certs\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.275656 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.275650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-home\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.275942 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.275669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-model-cache\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.275942 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.275692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgx5j\" (UniqueName: \"kubernetes.io/projected/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-kube-api-access-qgx5j\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.276095 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.276072 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-tmp-dir\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.276095 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.276089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-home\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.276193 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.276126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-dshm\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.276249 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.276192 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-model-cache\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.278220 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.278193 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-dshm\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.278405 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.278386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-tls-certs\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.284478 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.284457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgx5j\" (UniqueName: \"kubernetes.io/projected/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-kube-api-access-qgx5j\") pod \"auth-test-model-05e61d-kserve-86bf68887f-n29pl\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.313428 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.313408 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:10.652546 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:10.652522 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl"] Jul 22 17:22:10.655370 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:22:10.655344 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb078cd1d_aed9_4fdf_a2e1_7cd1c436593e.slice/crio-6d19a4eaadf9fd50acca6b97fe0afbd29d89a1dfa284e9c92fd0e3dfb6d7c5bb WatchSource:0}: Error finding container 6d19a4eaadf9fd50acca6b97fe0afbd29d89a1dfa284e9c92fd0e3dfb6d7c5bb: Status 404 returned error can't find the container with id 6d19a4eaadf9fd50acca6b97fe0afbd29d89a1dfa284e9c92fd0e3dfb6d7c5bb Jul 22 17:22:11.148341 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:11.148311 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" event={"ID":"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e","Type":"ContainerStarted","Data":"14f53f7427987e461b9d1d0b943d3456a2280a9d94e80aa81a8607fe15b06dbe"} Jul 22 17:22:11.148759 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:11.148351 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" event={"ID":"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e","Type":"ContainerStarted","Data":"6d19a4eaadf9fd50acca6b97fe0afbd29d89a1dfa284e9c92fd0e3dfb6d7c5bb"} Jul 22 17:22:11.164342 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:11.164288 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" podStartSLOduration=2.164268733 podStartE2EDuration="2.164268733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:22:11.163921466 +0000 UTC m=+2513.520317785" watchObservedRunningTime="2026-07-22 17:22:11.164268733 +0000 UTC m=+2513.520665052" Jul 22 17:22:13.708602 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:13.708566 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:22:20.314237 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:20.314203 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:20.314631 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:20.314250 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:20.326493 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:20.326469 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:21.202833 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:21.202807 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:22:35.889911 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:35.889881 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:22:53.399138 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:22:53.399093 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:23:09.997253 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:23:09.997218 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:23:27.396090 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:23:27.396049 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:23:45.198134 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:23:45.198052 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:24:03.485699 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:03.485662 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:24:19.616717 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:19.616686 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl"] Jul 22 17:24:21.999362 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:21.999329 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:24:34.637278 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:34.637212 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" podUID="b078cd1d-aed9-4fdf-a2e1-7cd1c436593e" containerName="main" containerID="cri-o://14f53f7427987e461b9d1d0b943d3456a2280a9d94e80aa81a8607fe15b06dbe" gracePeriod=45 Jul 22 17:24:34.826250 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:34.826215 2577 generic.go:356] "Generic (PLEG): container finished" podID="b078cd1d-aed9-4fdf-a2e1-7cd1c436593e" containerID="14f53f7427987e461b9d1d0b943d3456a2280a9d94e80aa81a8607fe15b06dbe" exitCode=0 Jul 22 17:24:34.826393 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:34.826262 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" event={"ID":"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e","Type":"ContainerDied","Data":"14f53f7427987e461b9d1d0b943d3456a2280a9d94e80aa81a8607fe15b06dbe"} Jul 22 17:24:34.884014 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:34.883989 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:24:35.026669 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.026639 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-tmp-dir\") pod \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " Jul 22 17:24:35.026849 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.026675 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-tls-certs\") pod \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " Jul 22 17:24:35.026849 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.026695 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgx5j\" (UniqueName: \"kubernetes.io/projected/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-kube-api-access-qgx5j\") pod \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " Jul 22 17:24:35.026849 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.026725 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-home\") pod \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " Jul 22 17:24:35.026849 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.026820 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-dshm\") pod \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " Jul 22 17:24:35.027092 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.026868 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-model-cache\") pod \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\" (UID: \"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e\") " Jul 22 17:24:35.027092 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.026961 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b078cd1d-aed9-4fdf-a2e1-7cd1c436593e" (UID: "b078cd1d-aed9-4fdf-a2e1-7cd1c436593e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:24:35.027092 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.027037 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-home" (OuterVolumeSpecName: "home") pod "b078cd1d-aed9-4fdf-a2e1-7cd1c436593e" (UID: "b078cd1d-aed9-4fdf-a2e1-7cd1c436593e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:24:35.027206 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.027088 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b078cd1d-aed9-4fdf-a2e1-7cd1c436593e" (UID: "b078cd1d-aed9-4fdf-a2e1-7cd1c436593e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:24:35.027251 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.027213 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-home\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:24:35.027251 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.027226 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-model-cache\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:24:35.027251 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.027237 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-tmp-dir\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:24:35.028846 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.028828 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b078cd1d-aed9-4fdf-a2e1-7cd1c436593e" (UID: "b078cd1d-aed9-4fdf-a2e1-7cd1c436593e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:24:35.029221 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.029198 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-dshm" (OuterVolumeSpecName: "dshm") pod "b078cd1d-aed9-4fdf-a2e1-7cd1c436593e" (UID: "b078cd1d-aed9-4fdf-a2e1-7cd1c436593e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:24:35.029316 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.029223 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-kube-api-access-qgx5j" (OuterVolumeSpecName: "kube-api-access-qgx5j") pod "b078cd1d-aed9-4fdf-a2e1-7cd1c436593e" (UID: "b078cd1d-aed9-4fdf-a2e1-7cd1c436593e"). InnerVolumeSpecName "kube-api-access-qgx5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:24:35.127619 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.127586 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-dshm\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:24:35.127619 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.127612 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-tls-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:24:35.127619 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.127622 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgx5j\" (UniqueName: \"kubernetes.io/projected/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e-kube-api-access-qgx5j\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:24:35.832393 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.832357 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" event={"ID":"b078cd1d-aed9-4fdf-a2e1-7cd1c436593e","Type":"ContainerDied","Data":"6d19a4eaadf9fd50acca6b97fe0afbd29d89a1dfa284e9c92fd0e3dfb6d7c5bb"} Jul 22 17:24:35.832813 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.832407 2577 scope.go:117] "RemoveContainer" containerID="14f53f7427987e461b9d1d0b943d3456a2280a9d94e80aa81a8607fe15b06dbe" Jul 22 17:24:35.832813 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.832406 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl" Jul 22 17:24:35.852680 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.852658 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl"] Jul 22 17:24:35.856402 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:35.856382 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-05e61d/auth-test-model-05e61d-kserve-86bf68887f-n29pl"] Jul 22 17:24:36.185873 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:36.185843 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b078cd1d-aed9-4fdf-a2e1-7cd1c436593e" path="/var/lib/kubelet/pods/b078cd1d-aed9-4fdf-a2e1-7cd1c436593e/volumes" Jul 22 17:24:36.191984 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:36.191962 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:24:49.592582 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:24:49.592545 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:25:05.890652 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:05.890614 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:25:30.195055 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:30.195021 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:25:38.704756 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.704704 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9"] Jul 22 17:25:38.705201 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.705086 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b078cd1d-aed9-4fdf-a2e1-7cd1c436593e" containerName="main" Jul 22 17:25:38.705201 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.705097 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b078cd1d-aed9-4fdf-a2e1-7cd1c436593e" containerName="main" Jul 22 17:25:38.705201 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.705171 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b078cd1d-aed9-4fdf-a2e1-7cd1c436593e" containerName="main" Jul 22 17:25:38.707222 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.707206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.710472 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.710440 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-629c13\"/\"kube-root-ca.crt\"" Jul 22 17:25:38.710472 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.710460 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-629c13\"/\"default-dockercfg-xqr92\"" Jul 22 17:25:38.710472 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.710465 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-629c13\"/\"rate-test-model-629c13-kserve-self-signed-certs\"" Jul 22 17:25:38.710691 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.710443 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-629c13\"/\"openshift-service-ca.crt\"" Jul 22 17:25:38.717262 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.717243 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9"] Jul 22 17:25:38.870255 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.870224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-tls-certs\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.870430 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.870271 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-dshm\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.870430 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.870306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-home\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.870430 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.870329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-model-cache\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.870430 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.870348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-tmp-dir\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.870430 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.870418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnwt\" (UniqueName: \"kubernetes.io/projected/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-kube-api-access-stnwt\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.971667 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.971576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-tls-certs\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.971667 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.971621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-dshm\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.971911 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.971677 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-home\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.971911 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.971714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-model-cache\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.971911 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.971756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-tmp-dir\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.971911 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.971795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnwt\" (UniqueName: \"kubernetes.io/projected/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-kube-api-access-stnwt\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.972179 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.972158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-model-cache\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.972251 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.972196 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-home\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.972251 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.972239 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-tmp-dir\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.974014 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.973998 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-dshm\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.974138 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.974111 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-tls-certs\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:38.980854 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:38.980826 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnwt\" (UniqueName: \"kubernetes.io/projected/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-kube-api-access-stnwt\") pod \"rate-test-model-629c13-kserve-8896b877f-58bx9\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:39.016353 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:39.016331 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:39.144032 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:39.144006 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9"] Jul 22 17:25:39.146956 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:25:39.146925 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb12f715_4abf_4fcb_b7d7_84b58b841fe1.slice/crio-dd2fc66b2bd7a41514e1fcdb8cb4a50bb7fafdccd41a20b5025779284796546d WatchSource:0}: Error finding container dd2fc66b2bd7a41514e1fcdb8cb4a50bb7fafdccd41a20b5025779284796546d: Status 404 returned error can't find the container with id dd2fc66b2bd7a41514e1fcdb8cb4a50bb7fafdccd41a20b5025779284796546d Jul 22 17:25:40.125500 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:40.125467 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" event={"ID":"fb12f715-4abf-4fcb-b7d7-84b58b841fe1","Type":"ContainerStarted","Data":"fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576"} Jul 22 17:25:40.125500 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:40.125506 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" event={"ID":"fb12f715-4abf-4fcb-b7d7-84b58b841fe1","Type":"ContainerStarted","Data":"dd2fc66b2bd7a41514e1fcdb8cb4a50bb7fafdccd41a20b5025779284796546d"} Jul 22 17:25:40.143608 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:40.143559 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" podStartSLOduration=2.143544704 podStartE2EDuration="2.143544704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:25:40.142149725 +0000 UTC m=+2722.498546056" watchObservedRunningTime="2026-07-22 17:25:40.143544704 +0000 UTC m=+2722.499941020" Jul 22 17:25:43.905554 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:43.905519 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:25:49.016979 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:49.016943 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:49.017363 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:49.016994 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:49.029285 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:49.029261 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:25:49.180027 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:25:49.180001 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:26:24.343885 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:26:24.343850 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:26:38.600665 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:26:38.600626 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:27:00.898425 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:00.898350 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:27:05.235162 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:05.235126 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9"] Jul 22 17:27:15.203760 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:15.203715 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:27:20.257048 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.257008 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" podUID="fb12f715-4abf-4fcb-b7d7-84b58b841fe1" containerName="main" containerID="cri-o://fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576" gracePeriod=45 Jul 22 17:27:20.507765 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.507688 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:27:20.586465 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.586429 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb12f715-4abf-4fcb-b7d7-84b58b841fe1" containerID="fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576" exitCode=0 Jul 22 17:27:20.586605 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.586473 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" Jul 22 17:27:20.586605 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.586501 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" event={"ID":"fb12f715-4abf-4fcb-b7d7-84b58b841fe1","Type":"ContainerDied","Data":"fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576"} Jul 22 17:27:20.586605 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.586537 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9" event={"ID":"fb12f715-4abf-4fcb-b7d7-84b58b841fe1","Type":"ContainerDied","Data":"dd2fc66b2bd7a41514e1fcdb8cb4a50bb7fafdccd41a20b5025779284796546d"} Jul 22 17:27:20.586605 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.586558 2577 scope.go:117] "RemoveContainer" containerID="fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576" Jul 22 17:27:20.594724 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.594704 2577 scope.go:117] "RemoveContainer" containerID="fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576" Jul 22 17:27:20.595006 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:27:20.594988 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576\": container with ID starting with fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576 not found: ID does not exist" containerID="fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576" Jul 22 17:27:20.595069 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.595020 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576"} err="failed to get container status \"fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576\": rpc error: code = NotFound desc = could not find container \"fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576\": container with ID starting with fbf9d6e6a3ba72337b75c39ea5ea4b2757ad04b34f84be9d84a35c3ca6db8576 not found: ID does not exist" Jul 22 17:27:20.646938 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.646917 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-dshm\") pod \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " Jul 22 17:27:20.647052 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.646991 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-model-cache\") pod \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " Jul 22 17:27:20.647052 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.647023 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-home\") pod \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " Jul 22 17:27:20.647159 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.647058 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-tls-certs\") pod \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " Jul 22 17:27:20.647159 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.647127 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-tmp-dir\") pod \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " Jul 22 17:27:20.647254 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.647160 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stnwt\" (UniqueName: \"kubernetes.io/projected/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-kube-api-access-stnwt\") pod \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\" (UID: \"fb12f715-4abf-4fcb-b7d7-84b58b841fe1\") " Jul 22 17:27:20.647303 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.647276 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb12f715-4abf-4fcb-b7d7-84b58b841fe1" (UID: "fb12f715-4abf-4fcb-b7d7-84b58b841fe1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:27:20.647424 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.647377 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb12f715-4abf-4fcb-b7d7-84b58b841fe1" (UID: "fb12f715-4abf-4fcb-b7d7-84b58b841fe1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:27:20.647498 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.647465 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-model-cache\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:20.647498 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.647482 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-tmp-dir\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:20.647607 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.647587 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-home" (OuterVolumeSpecName: "home") pod "fb12f715-4abf-4fcb-b7d7-84b58b841fe1" (UID: "fb12f715-4abf-4fcb-b7d7-84b58b841fe1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:27:20.649496 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.649476 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-dshm" (OuterVolumeSpecName: "dshm") pod "fb12f715-4abf-4fcb-b7d7-84b58b841fe1" (UID: "fb12f715-4abf-4fcb-b7d7-84b58b841fe1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:27:20.649496 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.649483 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb12f715-4abf-4fcb-b7d7-84b58b841fe1" (UID: "fb12f715-4abf-4fcb-b7d7-84b58b841fe1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:27:20.649624 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.649502 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-kube-api-access-stnwt" (OuterVolumeSpecName: "kube-api-access-stnwt") pod "fb12f715-4abf-4fcb-b7d7-84b58b841fe1" (UID: "fb12f715-4abf-4fcb-b7d7-84b58b841fe1"). InnerVolumeSpecName "kube-api-access-stnwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:27:20.748349 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.748310 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-home\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:20.748349 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.748346 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-tls-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:20.748349 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.748357 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stnwt\" (UniqueName: \"kubernetes.io/projected/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-kube-api-access-stnwt\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:20.748560 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.748366 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb12f715-4abf-4fcb-b7d7-84b58b841fe1-dshm\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:20.908255 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.908230 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9"] Jul 22 17:27:20.912662 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:20.912638 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-629c13/rate-test-model-629c13-kserve-8896b877f-58bx9"] Jul 22 17:27:22.185545 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:22.185511 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb12f715-4abf-4fcb-b7d7-84b58b841fe1" path="/var/lib/kubelet/pods/fb12f715-4abf-4fcb-b7d7-84b58b841fe1/volumes" Jul 22 17:27:39.085859 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:39.085822 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:27:47.242563 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:47.242529 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5"] Jul 22 17:27:47.242984 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:47.242786 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" podUID="28ceb759-f283-4f61-90d6-2ba297658818" containerName="istio-proxy" containerID="cri-o://30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a" gracePeriod=30 Jul 22 17:27:52.495803 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.495779 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:27:52.603300 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603228 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-credential-socket\") pod \"28ceb759-f283-4f61-90d6-2ba297658818\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " Jul 22 17:27:52.603300 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603276 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-istio-envoy\") pod \"28ceb759-f283-4f61-90d6-2ba297658818\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " Jul 22 17:27:52.603469 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603304 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-istio-data\") pod \"28ceb759-f283-4f61-90d6-2ba297658818\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " Jul 22 17:27:52.603469 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603328 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ceb759-f283-4f61-90d6-2ba297658818-istio-token\") pod \"28ceb759-f283-4f61-90d6-2ba297658818\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " Jul 22 17:27:52.603469 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603342 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ceb759-f283-4f61-90d6-2ba297658818-istiod-ca-cert\") pod \"28ceb759-f283-4f61-90d6-2ba297658818\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " Jul 22 17:27:52.603469 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603404 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-workload-certs\") pod \"28ceb759-f283-4f61-90d6-2ba297658818\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " Jul 22 17:27:52.603469 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603442 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m2wj\" (UniqueName: \"kubernetes.io/projected/28ceb759-f283-4f61-90d6-2ba297658818-kube-api-access-9m2wj\") pod \"28ceb759-f283-4f61-90d6-2ba297658818\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " Jul 22 17:27:52.603721 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603474 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-workload-socket\") pod \"28ceb759-f283-4f61-90d6-2ba297658818\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " Jul 22 17:27:52.603721 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603482 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "28ceb759-f283-4f61-90d6-2ba297658818" (UID: "28ceb759-f283-4f61-90d6-2ba297658818"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:27:52.603721 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603508 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ceb759-f283-4f61-90d6-2ba297658818-istio-podinfo\") pod \"28ceb759-f283-4f61-90d6-2ba297658818\" (UID: \"28ceb759-f283-4f61-90d6-2ba297658818\") " Jul 22 17:27:52.603944 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603722 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "28ceb759-f283-4f61-90d6-2ba297658818" (UID: "28ceb759-f283-4f61-90d6-2ba297658818"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:27:52.603944 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603763 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ceb759-f283-4f61-90d6-2ba297658818-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "28ceb759-f283-4f61-90d6-2ba297658818" (UID: "28ceb759-f283-4f61-90d6-2ba297658818"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:27:52.603944 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603855 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-credential-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:52.604094 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603874 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ceb759-f283-4f61-90d6-2ba297658818-istiod-ca-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:52.604094 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.604007 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-workload-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:52.604094 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.603983 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "28ceb759-f283-4f61-90d6-2ba297658818" (UID: "28ceb759-f283-4f61-90d6-2ba297658818"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:27:52.604464 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.604442 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-istio-data" (OuterVolumeSpecName: "istio-data") pod "28ceb759-f283-4f61-90d6-2ba297658818" (UID: "28ceb759-f283-4f61-90d6-2ba297658818"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:27:52.605620 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.605599 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ceb759-f283-4f61-90d6-2ba297658818-kube-api-access-9m2wj" (OuterVolumeSpecName: "kube-api-access-9m2wj") pod "28ceb759-f283-4f61-90d6-2ba297658818" (UID: "28ceb759-f283-4f61-90d6-2ba297658818"). InnerVolumeSpecName "kube-api-access-9m2wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:27:52.605732 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.605711 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28ceb759-f283-4f61-90d6-2ba297658818-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "28ceb759-f283-4f61-90d6-2ba297658818" (UID: "28ceb759-f283-4f61-90d6-2ba297658818"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 17:27:52.605920 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.605899 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ceb759-f283-4f61-90d6-2ba297658818-istio-token" (OuterVolumeSpecName: "istio-token") pod "28ceb759-f283-4f61-90d6-2ba297658818" (UID: "28ceb759-f283-4f61-90d6-2ba297658818"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:27:52.605920 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.605911 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "28ceb759-f283-4f61-90d6-2ba297658818" (UID: "28ceb759-f283-4f61-90d6-2ba297658818"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:27:52.705275 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.705248 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ceb759-f283-4f61-90d6-2ba297658818-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:52.705275 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.705273 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m2wj\" (UniqueName: \"kubernetes.io/projected/28ceb759-f283-4f61-90d6-2ba297658818-kube-api-access-9m2wj\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:52.705456 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.705289 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-workload-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:52.705456 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.705300 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ceb759-f283-4f61-90d6-2ba297658818-istio-podinfo\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:52.705456 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.705312 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-istio-envoy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:52.705456 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.705323 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ceb759-f283-4f61-90d6-2ba297658818-istio-data\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:27:52.727777 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.727725 2577 generic.go:356] "Generic (PLEG): container finished" podID="28ceb759-f283-4f61-90d6-2ba297658818" containerID="30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a" exitCode=0 Jul 22 17:27:52.727859 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.727769 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" event={"ID":"28ceb759-f283-4f61-90d6-2ba297658818","Type":"ContainerDied","Data":"30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a"} Jul 22 17:27:52.727859 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.727796 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" Jul 22 17:27:52.727859 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.727834 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5" event={"ID":"28ceb759-f283-4f61-90d6-2ba297658818","Type":"ContainerDied","Data":"542520be7993e69ba202de5f14b3a457c440e750a5c45243ef646797c5b5b50b"} Jul 22 17:27:52.728001 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.727865 2577 scope.go:117] "RemoveContainer" containerID="30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a" Jul 22 17:27:52.737097 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.737077 2577 scope.go:117] "RemoveContainer" containerID="4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44" Jul 22 17:27:52.744494 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.744476 2577 scope.go:117] "RemoveContainer" containerID="30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a" Jul 22 17:27:52.744749 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:27:52.744716 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a\": container with ID starting with 30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a not found: ID does not exist" containerID="30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a" Jul 22 17:27:52.744826 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.744804 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a"} err="failed to get container status \"30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a\": rpc error: code = NotFound desc = could not find container \"30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a\": container with ID starting with 30d139ac32289fce67645df5b758783848ec0086b0895064274b2da6c4e05d5a not found: ID does not exist" Jul 22 17:27:52.744826 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.744825 2577 scope.go:117] "RemoveContainer" containerID="4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44" Jul 22 17:27:52.745039 ip-10-0-136-75 kubenswrapper[2577]: E0722 17:27:52.745024 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44\": container with ID starting with 4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44 not found: ID does not exist" containerID="4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44" Jul 22 17:27:52.745088 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.745043 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44"} err="failed to get container status \"4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44\": rpc error: code = NotFound desc = could not find container \"4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44\": container with ID starting with 4b5fd09640bb92f710df3707eafc486b9387952a952ec780013e04653e6b1e44 not found: ID does not exist" Jul 22 17:27:52.749795 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.749775 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5"] Jul 22 17:27:52.754054 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:52.754034 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-629c13-openshift-default-67dc8c4c75-fs8k5"] Jul 22 17:27:54.186572 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:27:54.186541 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28ceb759-f283-4f61-90d6-2ba297658818" path="/var/lib/kubelet/pods/28ceb759-f283-4f61-90d6-2ba297658818/volumes" Jul 22 17:28:01.111976 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:01.111940 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nrqjk"] Jul 22 17:28:09.695972 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:09.695939 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk"] Jul 22 17:28:09.696383 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:09.696196 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" podUID="1413a620-eaf5-4d20-9263-279b44821241" containerName="istio-proxy" containerID="cri-o://50fb73641dc8f86d5903e2935974b553066ecf6d43c5ee3ffc9c42943e537bba" gracePeriod=30 Jul 22 17:28:14.829868 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:14.829832 2577 generic.go:356] "Generic (PLEG): container finished" podID="1413a620-eaf5-4d20-9263-279b44821241" containerID="50fb73641dc8f86d5903e2935974b553066ecf6d43c5ee3ffc9c42943e537bba" exitCode=0 Jul 22 17:28:14.830227 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:14.829897 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" event={"ID":"1413a620-eaf5-4d20-9263-279b44821241","Type":"ContainerDied","Data":"50fb73641dc8f86d5903e2935974b553066ecf6d43c5ee3ffc9c42943e537bba"} Jul 22 17:28:14.830227 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:14.829960 2577 scope.go:117] "RemoveContainer" containerID="fb9d9b18d4ce9e50119d8e387aff11036c6c58c14f1feffeeaf0f5b721cfcb05" Jul 22 17:28:14.951472 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:14.951450 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:28:15.090241 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090166 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-istio-envoy\") pod \"1413a620-eaf5-4d20-9263-279b44821241\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " Jul 22 17:28:15.090241 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090206 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-workload-certs\") pod \"1413a620-eaf5-4d20-9263-279b44821241\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " Jul 22 17:28:15.090241 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090240 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-credential-socket\") pod \"1413a620-eaf5-4d20-9263-279b44821241\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " Jul 22 17:28:15.090517 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090260 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcv9z\" (UniqueName: \"kubernetes.io/projected/1413a620-eaf5-4d20-9263-279b44821241-kube-api-access-tcv9z\") pod \"1413a620-eaf5-4d20-9263-279b44821241\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " Jul 22 17:28:15.090517 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090276 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1413a620-eaf5-4d20-9263-279b44821241-istiod-ca-cert\") pod \"1413a620-eaf5-4d20-9263-279b44821241\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " Jul 22 17:28:15.090517 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090307 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-istio-data\") pod \"1413a620-eaf5-4d20-9263-279b44821241\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " Jul 22 17:28:15.090517 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090327 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1413a620-eaf5-4d20-9263-279b44821241-istio-token\") pod \"1413a620-eaf5-4d20-9263-279b44821241\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " Jul 22 17:28:15.090517 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090346 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1413a620-eaf5-4d20-9263-279b44821241-istio-podinfo\") pod \"1413a620-eaf5-4d20-9263-279b44821241\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " Jul 22 17:28:15.090517 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090375 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-workload-socket\") pod \"1413a620-eaf5-4d20-9263-279b44821241\" (UID: \"1413a620-eaf5-4d20-9263-279b44821241\") " Jul 22 17:28:15.090517 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090495 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1413a620-eaf5-4d20-9263-279b44821241" (UID: "1413a620-eaf5-4d20-9263-279b44821241"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:28:15.090908 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090540 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1413a620-eaf5-4d20-9263-279b44821241" (UID: "1413a620-eaf5-4d20-9263-279b44821241"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:28:15.090908 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090655 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-credential-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:28:15.090908 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090675 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-workload-certs\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:28:15.090908 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090760 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1413a620-eaf5-4d20-9263-279b44821241" (UID: "1413a620-eaf5-4d20-9263-279b44821241"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:28:15.090908 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.090806 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1413a620-eaf5-4d20-9263-279b44821241-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1413a620-eaf5-4d20-9263-279b44821241" (UID: "1413a620-eaf5-4d20-9263-279b44821241"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:28:15.091288 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.091262 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-istio-data" (OuterVolumeSpecName: "istio-data") pod "1413a620-eaf5-4d20-9263-279b44821241" (UID: "1413a620-eaf5-4d20-9263-279b44821241"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:28:15.092457 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.092431 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1413a620-eaf5-4d20-9263-279b44821241-kube-api-access-tcv9z" (OuterVolumeSpecName: "kube-api-access-tcv9z") pod "1413a620-eaf5-4d20-9263-279b44821241" (UID: "1413a620-eaf5-4d20-9263-279b44821241"). InnerVolumeSpecName "kube-api-access-tcv9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:28:15.092595 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.092573 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1413a620-eaf5-4d20-9263-279b44821241-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1413a620-eaf5-4d20-9263-279b44821241" (UID: "1413a620-eaf5-4d20-9263-279b44821241"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 17:28:15.092595 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.092584 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1413a620-eaf5-4d20-9263-279b44821241-istio-token" (OuterVolumeSpecName: "istio-token") pod "1413a620-eaf5-4d20-9263-279b44821241" (UID: "1413a620-eaf5-4d20-9263-279b44821241"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:28:15.092759 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.092614 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1413a620-eaf5-4d20-9263-279b44821241" (UID: "1413a620-eaf5-4d20-9263-279b44821241"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 17:28:15.192025 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.191991 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-istio-envoy\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:28:15.192025 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.192020 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcv9z\" (UniqueName: \"kubernetes.io/projected/1413a620-eaf5-4d20-9263-279b44821241-kube-api-access-tcv9z\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:28:15.192025 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.192033 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1413a620-eaf5-4d20-9263-279b44821241-istiod-ca-cert\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:28:15.192255 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.192045 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-istio-data\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:28:15.192255 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.192055 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1413a620-eaf5-4d20-9263-279b44821241-istio-token\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:28:15.192255 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.192066 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1413a620-eaf5-4d20-9263-279b44821241-istio-podinfo\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:28:15.192255 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.192076 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1413a620-eaf5-4d20-9263-279b44821241-workload-socket\") on node \"ip-10-0-136-75.ec2.internal\" DevicePath \"\"" Jul 22 17:28:15.835376 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.835347 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" Jul 22 17:28:15.835778 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.835346 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk" event={"ID":"1413a620-eaf5-4d20-9263-279b44821241","Type":"ContainerDied","Data":"220b66b48b9b610d02ae6112d3c41c846972b81ec83cc01e4b8b819062cdd158"} Jul 22 17:28:15.835778 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.835506 2577 scope.go:117] "RemoveContainer" containerID="50fb73641dc8f86d5903e2935974b553066ecf6d43c5ee3ffc9c42943e537bba" Jul 22 17:28:15.857415 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.857390 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk"] Jul 22 17:28:15.862665 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:15.862632 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-05e61d-openshift-default-88b4b68dc-x4lzk"] Jul 22 17:28:16.055492 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:16.055471 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-c7l5p_c00ea5e5-2cb3-44a2-96b8-110b00d55b63/manager/0.log" Jul 22 17:28:16.185957 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:16.185933 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1413a620-eaf5-4d20-9263-279b44821241" path="/var/lib/kubelet/pods/1413a620-eaf5-4d20-9263-279b44821241/volumes" Jul 22 17:28:16.199209 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:16.199177 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-p4drm_7a4ea855-dfad-4558-94bd-090f17c49657/manager/0.log" Jul 22 17:28:16.876515 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:16.876492 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55b7bf7664-clb4j_06516966-2f01-4e52-8c83-65c2a860ab9c/copy-manifests/0.log" Jul 22 17:28:16.894998 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:16.894975 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55b7bf7664-clb4j_06516966-2f01-4e52-8c83-65c2a860ab9c/manager/0.log" Jul 22 17:28:19.213766 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:19.213723 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-p248h_a62547fc-ed8d-432d-950c-1be19941c257/manager/0.log" Jul 22 17:28:19.337422 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:19.337394 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-d2qkx_3287f248-c04f-4e37-a86b-edb3c10715e0/manager/0.log" Jul 22 17:28:19.587362 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:19.587289 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-z29kx_afe65379-c439-437c-be70-c07424eb758b/registry-server/0.log" Jul 22 17:28:19.725332 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:19.725307 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dc94s_4994b74d-442b-4da1-a066-42e5008afd9f/manager/0.log" Jul 22 17:28:19.859666 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:19.859602 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nrqjk_623d7e47-4b5b-40fb-a1cb-28e674413b68/limitador/0.log" Jul 22 17:28:19.987586 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:19.987541 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-72667_6e55b967-5a94-4fa9-9d2e-b6a33d84fbc0/manager/0.log" Jul 22 17:28:20.342790 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:20.342759 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447dqhhlp_8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7/istio-proxy/0.log" Jul 22 17:28:20.582585 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:20.582558 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c467c957f-7btr8_14769731-231c-4538-8692-265628d70b78/kube-auth-proxy/0.log" Jul 22 17:28:20.838465 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:20.838430 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-ghtbw_0045fa9b-2030-4758-b2ff-3b95cc8245e7/istio-proxy/1.log" Jul 22 17:28:22.051260 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:22.051230 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-gzzqg_e66d566b-e152-46eb-8ea9-7e14c003af16/main/0.log" Jul 22 17:28:27.456869 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.456833 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-52llv/must-gather-45xz5"] Jul 22 17:28:27.457231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457156 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1413a620-eaf5-4d20-9263-279b44821241" containerName="istio-proxy" Jul 22 17:28:27.457231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457166 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1413a620-eaf5-4d20-9263-279b44821241" containerName="istio-proxy" Jul 22 17:28:27.457231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457175 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb12f715-4abf-4fcb-b7d7-84b58b841fe1" containerName="main" Jul 22 17:28:27.457231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457180 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb12f715-4abf-4fcb-b7d7-84b58b841fe1" containerName="main" Jul 22 17:28:27.457231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457190 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ceb759-f283-4f61-90d6-2ba297658818" containerName="istio-proxy" Jul 22 17:28:27.457231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457195 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ceb759-f283-4f61-90d6-2ba297658818" containerName="istio-proxy" Jul 22 17:28:27.457231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457201 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ceb759-f283-4f61-90d6-2ba297658818" containerName="istio-proxy" Jul 22 17:28:27.457231 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457206 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ceb759-f283-4f61-90d6-2ba297658818" containerName="istio-proxy" Jul 22 17:28:27.457474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457263 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ceb759-f283-4f61-90d6-2ba297658818" containerName="istio-proxy" Jul 22 17:28:27.457474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457272 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb12f715-4abf-4fcb-b7d7-84b58b841fe1" containerName="main" Jul 22 17:28:27.457474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457278 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1413a620-eaf5-4d20-9263-279b44821241" containerName="istio-proxy" Jul 22 17:28:27.457474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457285 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1413a620-eaf5-4d20-9263-279b44821241" containerName="istio-proxy" Jul 22 17:28:27.457474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457337 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1413a620-eaf5-4d20-9263-279b44821241" containerName="istio-proxy" Jul 22 17:28:27.457474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457343 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1413a620-eaf5-4d20-9263-279b44821241" containerName="istio-proxy" Jul 22 17:28:27.457474 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.457389 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ceb759-f283-4f61-90d6-2ba297658818" containerName="istio-proxy" Jul 22 17:28:27.462090 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.462070 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-52llv/must-gather-45xz5" Jul 22 17:28:27.464540 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.464519 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-52llv\"/\"kube-root-ca.crt\"" Jul 22 17:28:27.464631 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.464518 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-52llv\"/\"default-dockercfg-rh2ng\"" Jul 22 17:28:27.465022 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.465007 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-52llv\"/\"openshift-service-ca.crt\"" Jul 22 17:28:27.469770 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.469728 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-52llv/must-gather-45xz5"] Jul 22 17:28:27.593594 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.593566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72-must-gather-output\") pod \"must-gather-45xz5\" (UID: \"d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72\") " pod="openshift-must-gather-52llv/must-gather-45xz5" Jul 22 17:28:27.593721 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.593600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvjm\" (UniqueName: \"kubernetes.io/projected/d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72-kube-api-access-dnvjm\") pod \"must-gather-45xz5\" (UID: \"d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72\") " pod="openshift-must-gather-52llv/must-gather-45xz5" Jul 22 17:28:27.694817 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.694784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72-must-gather-output\") pod \"must-gather-45xz5\" (UID: \"d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72\") " pod="openshift-must-gather-52llv/must-gather-45xz5" Jul 22 17:28:27.694944 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.694825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvjm\" (UniqueName: \"kubernetes.io/projected/d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72-kube-api-access-dnvjm\") pod \"must-gather-45xz5\" (UID: \"d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72\") " pod="openshift-must-gather-52llv/must-gather-45xz5" Jul 22 17:28:27.695085 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.695065 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72-must-gather-output\") pod \"must-gather-45xz5\" (UID: \"d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72\") " pod="openshift-must-gather-52llv/must-gather-45xz5" Jul 22 17:28:27.702440 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.702423 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvjm\" (UniqueName: \"kubernetes.io/projected/d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72-kube-api-access-dnvjm\") pod \"must-gather-45xz5\" (UID: \"d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72\") " pod="openshift-must-gather-52llv/must-gather-45xz5" Jul 22 17:28:27.771798 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.771728 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-52llv/must-gather-45xz5" Jul 22 17:28:27.905956 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.905933 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-52llv/must-gather-45xz5"] Jul 22 17:28:27.907992 ip-10-0-136-75 kubenswrapper[2577]: W0722 17:28:27.907953 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4b5646a_dd9f_47ad_a4c8_0a4aa9514e72.slice/crio-5201edf66dc3ac62d40f784450ef4c0ecce4959c072efb8c3a9a792ae0402d63 WatchSource:0}: Error finding container 5201edf66dc3ac62d40f784450ef4c0ecce4959c072efb8c3a9a792ae0402d63: Status 404 returned error can't find the container with id 5201edf66dc3ac62d40f784450ef4c0ecce4959c072efb8c3a9a792ae0402d63 Jul 22 17:28:27.909837 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:27.909822 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 17:28:28.897185 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:28.897144 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52llv/must-gather-45xz5" event={"ID":"d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72","Type":"ContainerStarted","Data":"9435a5defe61e1047f55df4e29c39ad1e85cf28cad766008b4aa1741b89efe04"} Jul 22 17:28:28.897654 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:28.897202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52llv/must-gather-45xz5" event={"ID":"d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72","Type":"ContainerStarted","Data":"a23587d09007b6e612abb6e3f135cca041e4ed77ad6beeb3a6c1d04067f2a4c4"} Jul 22 17:28:28.897654 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:28.897220 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52llv/must-gather-45xz5" event={"ID":"d4b5646a-dd9f-47ad-a4c8-0a4aa9514e72","Type":"ContainerStarted","Data":"5201edf66dc3ac62d40f784450ef4c0ecce4959c072efb8c3a9a792ae0402d63"} Jul 22 17:28:28.929935 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:28.929871 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-52llv/must-gather-45xz5" podStartSLOduration=1.227725956 podStartE2EDuration="1.929848658s" totalImagesPullingTime="702.122702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:28:28.92717247 +0000 UTC m=+2891.283568791" watchObservedRunningTime="2026-07-22 17:28:28.929848658 +0000 UTC m=+2891.286244978" Jul 22 17:28:30.322343 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:30.322312 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mvtsj_3d12a195-9286-4276-b484-9b5ea87c58fb/global-pull-secret-syncer/0.log" Jul 22 17:28:30.424086 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:30.424048 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d5smg_887dfeb5-3d3e-4a25-90a6-0f9ccbacce37/konnectivity-agent/0.log" Jul 22 17:28:30.527805 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:30.527731 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-75.ec2.internal_fccdd49421f01c5ce5ce4ca243020bbf/haproxy/0.log" Jul 22 17:28:34.413768 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:34.413727 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-p248h_a62547fc-ed8d-432d-950c-1be19941c257/manager/0.log" Jul 22 17:28:34.452534 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:34.452500 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-d2qkx_3287f248-c04f-4e37-a86b-edb3c10715e0/manager/0.log" Jul 22 17:28:34.581953 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:34.581917 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-z29kx_afe65379-c439-437c-be70-c07424eb758b/registry-server/0.log" Jul 22 17:28:34.803439 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:34.803351 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dc94s_4994b74d-442b-4da1-a066-42e5008afd9f/manager/0.log" Jul 22 17:28:34.842362 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:34.842285 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nrqjk_623d7e47-4b5b-40fb-a1cb-28e674413b68/limitador/0.log" Jul 22 17:28:34.947053 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:34.942047 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-72667_6e55b967-5a94-4fa9-9d2e-b6a33d84fbc0/manager/0.log" Jul 22 17:28:37.245665 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:37.245632 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfsnz_592621a7-e5b4-4f71-8641-20161ab4115c/node-exporter/0.log" Jul 22 17:28:37.271670 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:37.271575 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfsnz_592621a7-e5b4-4f71-8641-20161ab4115c/kube-rbac-proxy/0.log" Jul 22 17:28:37.295542 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:37.295438 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfsnz_592621a7-e5b4-4f71-8641-20161ab4115c/init-textfile/0.log" Jul 22 17:28:40.664071 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.664030 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26"] Jul 22 17:28:40.668541 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.668518 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.675651 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.675625 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26"] Jul 22 17:28:40.826076 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.826039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d13979f6-4274-4354-8bcd-6a0282c04955-proc\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.826326 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.826305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d13979f6-4274-4354-8bcd-6a0282c04955-lib-modules\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.826491 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.826473 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6hhs\" (UniqueName: \"kubernetes.io/projected/d13979f6-4274-4354-8bcd-6a0282c04955-kube-api-access-v6hhs\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.826679 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.826650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d13979f6-4274-4354-8bcd-6a0282c04955-podres\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.826841 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.826826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d13979f6-4274-4354-8bcd-6a0282c04955-sys\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.928203 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.928112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d13979f6-4274-4354-8bcd-6a0282c04955-proc\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.928203 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.928160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d13979f6-4274-4354-8bcd-6a0282c04955-lib-modules\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.928203 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.928192 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6hhs\" (UniqueName: \"kubernetes.io/projected/d13979f6-4274-4354-8bcd-6a0282c04955-kube-api-access-v6hhs\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.928468 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.928231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d13979f6-4274-4354-8bcd-6a0282c04955-podres\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.928468 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.928239 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d13979f6-4274-4354-8bcd-6a0282c04955-proc\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.928468 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.928264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d13979f6-4274-4354-8bcd-6a0282c04955-sys\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.928468 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.928371 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d13979f6-4274-4354-8bcd-6a0282c04955-podres\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.928468 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.928372 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d13979f6-4274-4354-8bcd-6a0282c04955-lib-modules\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.928468 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.928432 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d13979f6-4274-4354-8bcd-6a0282c04955-sys\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.938081 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.938051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6hhs\" (UniqueName: \"kubernetes.io/projected/d13979f6-4274-4354-8bcd-6a0282c04955-kube-api-access-v6hhs\") pod \"perf-node-gather-daemonset-xgd26\" (UID: \"d13979f6-4274-4354-8bcd-6a0282c04955\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:40.983114 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:40.983080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:41.152718 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:41.152667 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26"] Jul 22 17:28:41.982727 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:41.982689 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" event={"ID":"d13979f6-4274-4354-8bcd-6a0282c04955","Type":"ContainerStarted","Data":"db758cb2ae683a0ca499df51b5329aefd3a318f176fda4ee78a438d95f186d2f"} Jul 22 17:28:41.983135 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:41.982839 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:41.983135 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:41.982860 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" event={"ID":"d13979f6-4274-4354-8bcd-6a0282c04955","Type":"ContainerStarted","Data":"5d5266bac5f187df2363af98a836a2887755e728903081417ceb709842de70a4"} Jul 22 17:28:41.998890 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:41.998840 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" podStartSLOduration=1.998825034 podStartE2EDuration="1.998825034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 17:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:28:41.997013233 +0000 UTC m=+2904.353409555" watchObservedRunningTime="2026-07-22 17:28:41.998825034 +0000 UTC m=+2904.355221371" Jul 22 17:28:42.513440 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:42.513414 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xl265_146e6ed1-00ed-473d-b3ef-9f50e261b7bf/dns/0.log" Jul 22 17:28:42.533141 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:42.533109 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xl265_146e6ed1-00ed-473d-b3ef-9f50e261b7bf/kube-rbac-proxy/0.log" Jul 22 17:28:42.600674 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:42.600645 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pp7b8_a03e0387-4890-4dec-9ac2-25b236bbdf69/dns-node-resolver/0.log" Jul 22 17:28:43.138731 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:43.138699 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6645b5458d-ghrb4_ffbcd5b9-3ad3-4d66-a2cb-85c9cded2cc7/registry/0.log" Jul 22 17:28:43.186350 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:43.186324 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hxj9q_407e45e8-f090-4b24-aa8d-53fabbe789ac/node-ca/0.log" Jul 22 17:28:44.072201 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:44.072167 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447dqhhlp_8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7/istio-proxy/0.log" Jul 22 17:28:44.214620 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:44.214592 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c467c957f-7btr8_14769731-231c-4538-8692-265628d70b78/kube-auth-proxy/0.log" Jul 22 17:28:44.270136 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:44.270088 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-ghtbw_0045fa9b-2030-4758-b2ff-3b95cc8245e7/istio-proxy/0.log" Jul 22 17:28:44.338619 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:44.338535 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-ghtbw_0045fa9b-2030-4758-b2ff-3b95cc8245e7/istio-proxy/1.log" Jul 22 17:28:45.138509 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:45.138454 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m89nb_3786152c-b59d-4c45-a6e1-a3b3ad670056/serve-healthcheck-canary/0.log" Jul 22 17:28:45.614778 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:45.614724 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bc66d_fb95755e-21d5-435c-9b16-eeb89295e8d0/kube-rbac-proxy/0.log" Jul 22 17:28:45.635415 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:45.635380 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bc66d_fb95755e-21d5-435c-9b16-eeb89295e8d0/exporter/0.log" Jul 22 17:28:45.658450 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:45.658422 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bc66d_fb95755e-21d5-435c-9b16-eeb89295e8d0/extractor/0.log" Jul 22 17:28:47.830529 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:47.830495 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-c7l5p_c00ea5e5-2cb3-44a2-96b8-110b00d55b63/manager/0.log" Jul 22 17:28:48.002676 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:48.002649 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-xgd26" Jul 22 17:28:48.227442 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:48.227409 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-p4drm_7a4ea855-dfad-4558-94bd-090f17c49657/manager/0.log" Jul 22 17:28:48.625204 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:48.625128 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55b7bf7664-clb4j_06516966-2f01-4e52-8c83-65c2a860ab9c/manager/0.log" Jul 22 17:28:48.644106 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:48.644083 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55b7bf7664-clb4j_06516966-2f01-4e52-8c83-65c2a860ab9c/copy-manifests/0.log" Jul 22 17:28:50.685512 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:50.685473 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447dqhhlp_8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7/istio-proxy/0.log" Jul 22 17:28:50.820142 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:50.820115 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c467c957f-7btr8_14769731-231c-4538-8692-265628d70b78/kube-auth-proxy/0.log" Jul 22 17:28:50.863605 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:50.863472 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-ghtbw_0045fa9b-2030-4758-b2ff-3b95cc8245e7/istio-proxy/0.log" Jul 22 17:28:50.923533 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:50.923502 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-ghtbw_0045fa9b-2030-4758-b2ff-3b95cc8245e7/istio-proxy/1.log" Jul 22 17:28:52.118178 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:52.118105 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447dqhhlp_8f9ceec5-7144-49f6-8cf7-9cb2ec07a0f7/istio-proxy/0.log" Jul 22 17:28:52.247482 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:52.247450 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c467c957f-7btr8_14769731-231c-4538-8692-265628d70b78/kube-auth-proxy/0.log" Jul 22 17:28:52.288290 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:52.288255 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-ghtbw_0045fa9b-2030-4758-b2ff-3b95cc8245e7/istio-proxy/0.log" Jul 22 17:28:52.343348 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:52.343309 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-ghtbw_0045fa9b-2030-4758-b2ff-3b95cc8245e7/istio-proxy/1.log" Jul 22 17:28:55.165344 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:55.165304 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-bh8r9_bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b/migrator/0.log" Jul 22 17:28:55.208099 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:55.208067 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-bh8r9_bfb8050e-bdd7-4ff8-9fe7-d927e4f5ef5b/graceful-termination/0.log" Jul 22 17:28:56.827204 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:56.827179 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w9vdr_9fd9a27a-b50a-4409-9cd4-690c8b6cf906/kube-multus-additional-cni-plugins/0.log" Jul 22 17:28:56.849554 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:56.849526 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w9vdr_9fd9a27a-b50a-4409-9cd4-690c8b6cf906/egress-router-binary-copy/0.log" Jul 22 17:28:56.873087 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:56.873063 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w9vdr_9fd9a27a-b50a-4409-9cd4-690c8b6cf906/cni-plugins/0.log" Jul 22 17:28:56.893112 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:56.893087 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w9vdr_9fd9a27a-b50a-4409-9cd4-690c8b6cf906/bond-cni-plugin/0.log" Jul 22 17:28:56.914788 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:56.914764 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w9vdr_9fd9a27a-b50a-4409-9cd4-690c8b6cf906/routeoverride-cni/0.log" Jul 22 17:28:56.934648 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:56.934624 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w9vdr_9fd9a27a-b50a-4409-9cd4-690c8b6cf906/whereabouts-cni-bincopy/0.log" Jul 22 17:28:56.956513 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:56.956491 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w9vdr_9fd9a27a-b50a-4409-9cd4-690c8b6cf906/whereabouts-cni/0.log" Jul 22 17:28:57.000921 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:57.000888 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fkmdr_dfd7f86a-055f-4b35-b7ba-52acb954481a/kube-multus/0.log" Jul 22 17:28:57.163478 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:57.163451 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zx8w2_659ef35b-7c38-49f0-bd4a-a8c99e6f2f38/network-metrics-daemon/0.log" Jul 22 17:28:57.180882 ip-10-0-136-75 kubenswrapper[2577]: I0722 17:28:57.180859 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zx8w2_659ef35b-7c38-49f0-bd4a-a8c99e6f2f38/kube-rbac-proxy/0.log"