Jul 24 00:13:43.290254 ip-10-0-134-162 systemd[1]: Starting Kubernetes Kubelet... Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.752208 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757608 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757625 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757629 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757635 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757638 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757641 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757643 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 00:13:43.838186 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757648 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757651 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757653 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757656 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757659 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757662 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757664 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757677 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757680 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757683 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757686 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757689 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757692 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757694 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757697 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757700 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757703 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757724 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757727 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757730 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 00:13:44.089691 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757733 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 00:13:43.838549 ip-10-0-134-162 systemd[1]: Started Kubernetes Kubelet. Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757736 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757739 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757742 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757749 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757752 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757755 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757757 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757760 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757763 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757765 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757768 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757771 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757774 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757776 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757779 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757782 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757784 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757787 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 00:13:44.097859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757790 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757795 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757800 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757804 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757808 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757812 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757815 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757818 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757822 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757825 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757827 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757830 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757832 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757835 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757839 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757842 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757846 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757850 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757853 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 00:13:44.100259 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757857 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757861 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757867 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757871 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757876 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757880 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757885 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757891 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757896 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757901 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757905 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757909 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757913 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757918 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757923 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757927 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757931 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757935 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757940 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 00:13:44.104861 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757959 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757964 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757968 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757972 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757976 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757980 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757983 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757987 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757991 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.757995 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758000 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758004 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758007 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758011 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758015 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758755 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758762 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758765 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758768 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758770 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 00:13:44.106787 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758773 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758776 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758778 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758782 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758784 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758788 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758791 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758794 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758796 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758799 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758802 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758805 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758808 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758811 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758813 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758816 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758819 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758822 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758824 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758827 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 00:13:44.109021 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758829 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758832 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758835 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758838 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758841 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758845 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758848 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758851 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758853 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758856 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758859 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758862 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758865 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758868 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758870 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758873 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758876 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758878 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758881 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758883 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 00:13:44.110317 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758886 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758888 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758891 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758894 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758897 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758900 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758917 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758919 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758922 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758925 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758927 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758930 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758933 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758937 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758940 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758942 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758965 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758968 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758971 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 00:13:44.111051 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758974 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758976 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758979 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758982 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758984 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758987 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758990 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758992 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758995 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758997 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.758999 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759004 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759007 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759010 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759012 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759015 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759019 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759023 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759027 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 00:13:44.111707 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759032 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759035 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759038 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759042 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759044 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759047 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759050 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759054 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759057 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759059 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759062 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759064 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759067 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759069 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759072 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759074 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759531 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759538 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759543 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 00:13:44.112327 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759546 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759550 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759554 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759557 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759560 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759563 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759566 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759569 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759571 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759576 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759579 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759582 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759584 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759588 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759591 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759593 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759596 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759598 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759601 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 00:13:44.112876 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759604 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759608 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759610 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759613 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759616 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759618 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759620 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759623 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759626 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759628 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759631 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759634 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759637 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759639 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759642 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759644 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759647 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759649 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759652 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759654 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 00:13:44.113491 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759657 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759659 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759663 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759665 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759668 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759670 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759673 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759676 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759678 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759681 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759683 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759686 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759688 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759692 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759695 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759697 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759699 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759702 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759704 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759707 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 00:13:44.114153 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759709 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759712 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759714 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759716 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759719 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759721 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759724 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759726 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759730 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759733 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759735 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759737 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759740 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759742 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759746 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759748 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759751 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759753 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759756 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 00:13:44.114785 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759759 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759763 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759765 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759767 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759770 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759773 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759776 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759779 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759781 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759783 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759786 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759789 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759791 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759794 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759796 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759798 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759801 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.759803 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759879 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759889 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759898 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 00:13:44.115387 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759904 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759911 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759915 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759922 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759926 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759929 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759936 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759939 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759942 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759962 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759965 2576 flags.go:64] FLAG: --cgroup-root="" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759968 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759971 2576 flags.go:64] FLAG: --client-ca-file="" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759976 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759979 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759984 2576 flags.go:64] FLAG: --cluster-domain="" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759987 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759992 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759995 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.759999 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760003 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760006 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760009 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 00:13:44.115996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760012 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760016 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760019 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760022 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760025 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760028 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760032 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760035 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760039 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760042 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760045 2576 flags.go:64] FLAG: --enable-server="true" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760048 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760052 2576 flags.go:64] FLAG: --event-burst="100" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760055 2576 flags.go:64] FLAG: --event-qps="50" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760058 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760062 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760065 2576 flags.go:64] FLAG: --eviction-hard="" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760069 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760072 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760075 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760078 2576 flags.go:64] FLAG: --eviction-soft="" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760081 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760083 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760087 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760090 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 00:13:44.116576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760093 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760097 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760100 2576 flags.go:64] FLAG: --feature-gates="" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760104 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760107 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760110 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760113 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760116 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760119 2576 flags.go:64] FLAG: --help="false" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760122 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760125 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760128 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760131 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760134 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760137 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760140 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760143 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760146 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760149 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760151 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760154 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760157 2576 flags.go:64] FLAG: --kube-reserved="" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760162 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760164 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 00:13:44.117299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760167 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760170 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760173 2576 flags.go:64] FLAG: --lock-file="" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760176 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760179 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760182 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760187 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760197 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760200 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760204 2576 flags.go:64] FLAG: --logging-format="text" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760207 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760210 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760213 2576 flags.go:64] FLAG: --manifest-url="" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760216 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760220 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760223 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760227 2576 flags.go:64] FLAG: --max-pods="110" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760230 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760233 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760236 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760238 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760241 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760244 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760248 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760255 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 00:13:44.120331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760258 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760261 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760264 2576 flags.go:64] FLAG: --pod-cidr="" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760266 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760272 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760275 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760279 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760282 2576 flags.go:64] FLAG: --port="10250" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760285 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760288 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b4d18e0a742dabc3" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760291 2576 flags.go:64] FLAG: --qos-reserved="" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760302 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760305 2576 flags.go:64] FLAG: --register-node="true" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760308 2576 flags.go:64] FLAG: --register-with-taints="" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760312 2576 flags.go:64] FLAG: --registry-burst="10" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760321 2576 flags.go:64] FLAG: --registry-qps="5" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760326 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760329 2576 flags.go:64] FLAG: --reserved-memory="" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760333 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760335 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760338 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760341 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760344 2576 flags.go:64] FLAG: --runonce="false" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760347 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760350 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 00:13:44.121149 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760353 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760355 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760358 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760361 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760364 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760368 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760370 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760373 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760376 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760379 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760382 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760385 2576 flags.go:64] FLAG: --system-cgroups="" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760387 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760395 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760398 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760400 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760407 2576 flags.go:64] FLAG: --tls-min-version="" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760410 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760413 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760415 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760418 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760421 2576 flags.go:64] FLAG: --v="2" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760425 2576 flags.go:64] FLAG: --version="false" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760432 2576 flags.go:64] FLAG: --vmodule="" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760436 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 00:13:44.124238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.760439 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760562 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760566 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760569 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760572 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760575 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760578 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760580 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760583 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760585 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760588 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760590 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760593 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760595 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760598 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760600 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760603 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760605 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760608 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760610 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 00:13:44.124908 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760614 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760616 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760619 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760622 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760624 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760626 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760629 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760632 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760634 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760637 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760641 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760644 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760647 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760649 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760652 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760654 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760657 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760659 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760662 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760664 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 00:13:44.287415 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760667 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760669 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760672 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760674 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760677 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760679 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760682 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760684 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760687 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760690 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760694 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760696 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760700 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760704 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760708 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760710 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760713 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760716 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760719 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 00:13:44.288347 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760722 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760724 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760727 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760731 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760734 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760737 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760740 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760742 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760745 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760747 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760750 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760752 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760756 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760758 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760761 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760763 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760766 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760768 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760770 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760773 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 00:13:44.288874 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760776 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760778 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760781 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760783 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760786 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760789 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760792 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760794 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760797 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760799 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760803 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760806 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760809 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760812 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760815 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760819 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760822 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760824 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760828 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760830 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 00:13:44.289451 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.760833 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.761480 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.769013 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.769034 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.769054 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.769060 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769120 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769125 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769128 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769131 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769134 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769138 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769142 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769146 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 00:13:44.290048 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769149 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769152 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769155 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769157 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769160 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769163 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769166 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769168 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769171 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769174 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769176 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769179 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769182 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769184 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769187 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769190 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769196 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769199 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769202 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769206 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769209 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 00:13:44.290482 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769212 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769215 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769217 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769220 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769222 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769225 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769227 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769231 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769233 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769235 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769238 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769240 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769243 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769246 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769248 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769250 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769253 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769255 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769258 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 00:13:44.291054 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769261 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769263 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769267 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769270 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769274 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769277 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769282 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769285 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769287 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769290 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769293 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769297 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769299 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769302 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769305 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769308 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769310 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769313 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769316 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 00:13:44.291567 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769318 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769320 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769323 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769326 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769329 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769333 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769336 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769338 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769341 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769343 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769345 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769348 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769350 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769353 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769355 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769358 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769360 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769363 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769366 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769371 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 00:13:44.292072 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769374 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769376 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769379 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769382 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769385 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769387 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769390 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769393 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769396 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769398 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769401 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769403 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.769408 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769515 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769523 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 00:13:44.292615 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769526 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769529 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769532 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769535 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769538 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769541 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769543 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769546 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769549 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769552 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769556 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769559 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769561 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769564 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769567 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769570 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769573 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769576 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769579 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 00:13:44.293038 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769581 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769584 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769587 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769590 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769592 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769595 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769597 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769599 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769602 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769604 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769607 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769610 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769614 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769617 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769619 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769622 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769625 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769628 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769630 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769633 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 00:13:44.293565 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769636 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769638 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769640 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769643 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769646 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769648 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769650 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769653 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769655 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769661 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769664 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769673 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769677 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769680 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769683 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769686 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769689 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769691 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769694 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769696 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 00:13:44.294137 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769699 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769702 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769724 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769728 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769732 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769734 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769737 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769740 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769742 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769745 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769747 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769750 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769753 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769755 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769758 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769760 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769763 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769765 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769768 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769770 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 00:13:44.294685 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769773 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769776 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769779 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769782 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769784 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769787 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769789 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769792 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769795 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769797 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769800 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769803 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769805 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769808 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769812 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769815 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769820 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 00:13:44.295284 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:43.769823 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 00:13:44.295744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.769828 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 00:13:44.295744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.769969 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 00:13:44.295744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.772225 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 00:13:44.295744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.774283 2576 server.go:1019] "Starting client certificate rotation" Jul 24 00:13:44.295744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.774401 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 00:13:44.295744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.774437 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 00:13:44.295744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.777638 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 00:13:44.295744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.785556 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 00:13:44.295744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.798160 2576 log.go:25] "Validated CRI v1 runtime API" Jul 24 00:13:44.295744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.802545 2576 log.go:25] "Validated CRI v1 image API" Jul 24 00:13:44.295744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.803421 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 00:13:44.295744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.806634 2576 fs.go:135] Filesystem UUIDs: map[41ad37a6-f66d-41bc-9267-094222b0f94d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a2ca826d-60bf-421f-af76-a0fbcbe94a29:/dev/nvme0n1p3] Jul 24 00:13:44.296094 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.806656 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 00:13:44.296094 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.808427 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.811275 2576 manager.go:216] Machine: {Timestamp:2026-07-24 00:13:43.810775311 +0000 UTC m=+0.383872161 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3192724 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29456d661f05c712627ff6b909ed8d SystemUUID:ec29456d-661f-05c7-1262-7ff6b909ed8d BootID:54a8af8a-c8f6-40d0-80a6-ceefe1657353 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3d:2f:f6:ed:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3d:2f:f6:ed:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:14:c6:82:4d:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.811380 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.811464 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.814346 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.814372 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-162.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.815634 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.815647 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.815655 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.816444 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.817355 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.819624 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.819758 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.819786 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.819796 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.819812 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.819829 2576 kubelet.go:398] "Adding apiserver pod source" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.819841 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 00:13:44.296301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.820271 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.820293 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.823802 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.826420 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.826449 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.828123 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.828136 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.828142 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.828148 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.828160 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.828168 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.828178 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.828183 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.828190 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.828195 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.828204 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.828209 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.829157 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.829163 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.830983 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dxjsf" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.831985 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-162.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:43.832053 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:43.832363 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-162.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.835963 2576 server.go:1268] "Started kubelet" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.836737 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.838097 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.838213 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.838846 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dxjsf" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.839738 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.840280 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:43.845049 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:43.846318 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-162.ec2.internal.18c5117589604a73 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-162.ec2.internal,UID:ip-10-0-134-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-162.ec2.internal,},FirstTimestamp:2026-07-24 00:13:43.835904627 +0000 UTC m=+0.409001420,LastTimestamp:2026-07-24 00:13:43.835904627 +0000 UTC m=+0.409001420,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-162.ec2.internal,}" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.849683 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.849698 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:43.849773 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.850380 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.850380 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.850402 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.850469 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.850477 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 24 00:13:44.296832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.850554 2576 factory.go:55] Registering systemd factory Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.850571 2576 factory.go:223] Registration of the systemd container factory successfully Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.850827 2576 factory.go:153] Registering CRI-O factory Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.850843 2576 factory.go:223] Registration of the crio container factory successfully Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.850894 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.850922 2576 factory.go:103] Registering Raw factory Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.850937 2576 manager.go:1196] Started watching for new ooms in manager Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.851375 2576 manager.go:349] Starting recovery of all containers Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.859321 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.861181 2576 manager.go:354] Recovery completed Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:43.861658 2576 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-162.ec2.internal\" not found" node="ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.866012 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.868227 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.868256 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.868269 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.868897 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.868908 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.868908 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.868983 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.869656 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.869691 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.869747 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.873107 2576 policy_none.go:49] "None policy: Start" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.873136 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.873148 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.874432 2576 policy_none.go:47] "Start" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:43.950148 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.952602 2576 manager.go:337] "Starting Device Plugin manager" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:43.952633 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.952643 2576 server.go:87] "Starting device plugin registration server" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.952891 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.952904 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.953030 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.953109 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.953117 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:43.953545 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:43.953594 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.993830 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.995075 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.995112 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.995135 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:43.995169 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:43.997159 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.053111 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.054127 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.054157 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.054169 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.054195 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.062703 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.302726 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.062732 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-162.ec2.internal\": node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.072880 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.095832 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-162.ec2.internal"] Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.096183 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.098304 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.098344 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.098354 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.099564 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.099675 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.099703 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.100425 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.100448 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.100464 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.100471 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.100481 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.100483 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.101623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.101655 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.102898 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.102939 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.103045 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.137921 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-162.ec2.internal\" not found" node="ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.142446 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-162.ec2.internal\" not found" node="ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.153101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/893e9e357457367d0e01aa3fbac60606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal\" (UID: \"893e9e357457367d0e01aa3fbac60606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.153130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/893e9e357457367d0e01aa3fbac60606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal\" (UID: \"893e9e357457367d0e01aa3fbac60606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.153148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/facb852d64cc5d31f3a76f06db558e9e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-162.ec2.internal\" (UID: \"facb852d64cc5d31f3a76f06db558e9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.173502 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.253479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/893e9e357457367d0e01aa3fbac60606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal\" (UID: \"893e9e357457367d0e01aa3fbac60606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.451969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.253496 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/893e9e357457367d0e01aa3fbac60606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal\" (UID: \"893e9e357457367d0e01aa3fbac60606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.454312 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.253522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/893e9e357457367d0e01aa3fbac60606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal\" (UID: \"893e9e357457367d0e01aa3fbac60606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.454312 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.253539 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/facb852d64cc5d31f3a76f06db558e9e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-162.ec2.internal\" (UID: \"facb852d64cc5d31f3a76f06db558e9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.454312 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.253576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/facb852d64cc5d31f3a76f06db558e9e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-162.ec2.internal\" (UID: \"facb852d64cc5d31f3a76f06db558e9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.454312 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.253616 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/893e9e357457367d0e01aa3fbac60606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal\" (UID: \"893e9e357457367d0e01aa3fbac60606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.454312 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.274579 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:44.454312 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.375373 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:44.454312 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.440557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.454312 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.445661 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-162.ec2.internal" Jul 24 00:13:44.576276 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.476272 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:44.578884 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.576772 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:44.758077 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.677299 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:44.935938 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.772772 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 00:13:44.935938 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.772908 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 00:13:44.935938 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.772937 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 00:13:44.935938 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.777916 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:44.935938 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.847382 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 00:08:44 +0000 UTC" deadline="2028-02-17 01:07:39.990552689 +0000 UTC" Jul 24 00:13:44.935938 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:44.847408 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13752h53m55.143147709s" Jul 24 00:13:44.935938 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.878546 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:45.067434 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:44.979442 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:45.210815 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:45.080147 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:45.210815 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.123802 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 00:13:45.210815 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:45.129541 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacb852d64cc5d31f3a76f06db558e9e.slice/crio-3bfb49c19762aace4b2d4deab6a2dbe8a97d0d18c430218f868813e520c49e74 WatchSource:0}: Error finding container 3bfb49c19762aace4b2d4deab6a2dbe8a97d0d18c430218f868813e520c49e74: Status 404 returned error can't find the container with id 3bfb49c19762aace4b2d4deab6a2dbe8a97d0d18c430218f868813e520c49e74 Jul 24 00:13:45.210815 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:45.130988 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893e9e357457367d0e01aa3fbac60606.slice/crio-f640553b3e4f9eab474e9753b354e62047780321b91b5783e9154e4ff08f893d WatchSource:0}: Error finding container f640553b3e4f9eab474e9753b354e62047780321b91b5783e9154e4ff08f893d: Status 404 returned error can't find the container with id f640553b3e4f9eab474e9753b354e62047780321b91b5783e9154e4ff08f893d Jul 24 00:13:45.210815 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.135916 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 00:13:45.210815 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:45.180645 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-162.ec2.internal\" not found" Jul 24 00:13:45.382625 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.242595 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 00:13:45.382625 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.249884 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" Jul 24 00:13:45.382625 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.261299 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 00:13:45.382625 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.262372 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-162.ec2.internal" Jul 24 00:13:45.382625 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.270598 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.821116 2576 apiserver.go:52] "Watching apiserver" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.828199 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.828626 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-162.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p","openshift-image-registry/node-ca-h6xgq","openshift-network-operator/iptables-alerter-5lnr9","openshift-ovn-kubernetes/ovnkube-node-dtgbd","kube-system/global-pull-secret-syncer-vncds","openshift-cluster-node-tuning-operator/tuned-xxscj","openshift-dns/node-resolver-8lgmt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal","openshift-multus/multus-additional-cni-plugins-vt9rk","openshift-multus/multus-jzwjn","openshift-multus/network-metrics-daemon-9dmnl","openshift-network-diagnostics/network-check-target-xnrdp","kube-system/konnectivity-agent-mw2d4"] Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.844411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.846888 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:45.846977 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.847702 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.847797 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.848000 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j92ws\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.849454 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h6xgq" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.850085 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.851882 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.851882 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.852161 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zlm2t\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.852468 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.852692 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5lnr9" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.854722 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.854946 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.855385 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.855476 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.855568 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-42ml2\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.857319 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:45.857388 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.857627 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.857827 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.857949 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.858125 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.858246 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.858258 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6r6tj\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.858243 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 00:13:45.878198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.859840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mw2d4" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.860727 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.863359 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-run\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af9ad08-fbcf-4275-97c8-d7c990f050e3-iptables-alerter-script\") pod \"iptables-alerter-5lnr9\" (UID: \"3af9ad08-fbcf-4275-97c8-d7c990f050e3\") " pod="openshift-network-operator/iptables-alerter-5lnr9" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864171 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af9ad08-fbcf-4275-97c8-d7c990f050e3-host-slash\") pod \"iptables-alerter-5lnr9\" (UID: \"3af9ad08-fbcf-4275-97c8-d7c990f050e3\") " pod="openshift-network-operator/iptables-alerter-5lnr9" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnlvf\" (UniqueName: \"kubernetes.io/projected/3af9ad08-fbcf-4275-97c8-d7c990f050e3-kube-api-access-cnlvf\") pod \"iptables-alerter-5lnr9\" (UID: \"3af9ad08-fbcf-4275-97c8-d7c990f050e3\") " pod="openshift-network-operator/iptables-alerter-5lnr9" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-modprobe-d\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-sysctl-d\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-host\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed8e9626-cece-4f7e-a51a-8a6a82082f09-tmp\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdnkk\" (UniqueName: \"kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk\") pod \"network-check-target-xnrdp\" (UID: \"84eade1e-4ce4-4c40-84f7-85ea4dfae29b\") " pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1261605-62c9-473d-bf74-156b31afd9c6-serviceca\") pod \"node-ca-h6xgq\" (UID: \"f1261605-62c9-473d-bf74-156b31afd9c6\") " pod="openshift-image-registry/node-ca-h6xgq" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-sys\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-systemd\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864934 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-tuned\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27hv\" (UniqueName: \"kubernetes.io/projected/ed8e9626-cece-4f7e-a51a-8a6a82082f09-kube-api-access-d27hv\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.864990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1261605-62c9-473d-bf74-156b31afd9c6-host\") pod \"node-ca-h6xgq\" (UID: \"f1261605-62c9-473d-bf74-156b31afd9c6\") " pod="openshift-image-registry/node-ca-h6xgq" Jul 24 00:13:45.881552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.865008 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.865040 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2vs4\" (UniqueName: \"kubernetes.io/projected/f1261605-62c9-473d-bf74-156b31afd9c6-kube-api-access-x2vs4\") pod \"node-ca-h6xgq\" (UID: \"f1261605-62c9-473d-bf74-156b31afd9c6\") " pod="openshift-image-registry/node-ca-h6xgq" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.865520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-sysconfig\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.865553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-kubernetes\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.865577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-sysctl-conf\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.865622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-lib-modules\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.865651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-var-lib-kubelet\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.866128 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9fc75\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.867034 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.867542 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8lgmt" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.869373 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.869839 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.869862 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.870019 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.870312 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c4vct\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.870444 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8l6np\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.870733 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.870929 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.872708 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.872717 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.872888 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.872956 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.873208 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.873299 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xd2hx\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.874750 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.877337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:45.877408 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.877820 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fkmfl\"" Jul 24 00:13:45.882910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.877892 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 00:13:45.887652 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.886146 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ftw67" Jul 24 00:13:45.899994 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.895437 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ftw67" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.951407 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-run-k8s-cni-cncf-io\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-etc-kubernetes\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-host\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6ffc7b0-5f19-4258-947c-95b01d7cb955-cni-binary-copy\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966320 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-kubelet\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-node-log\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-sys\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d92505bf-3059-4df8-8286-9cbd3bf693c1-cnibin\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d92505bf-3059-4df8-8286-9cbd3bf693c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-var-lib-cni-bin\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966481 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-run-systemd\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e6ffc7b0-5f19-4258-947c-95b01d7cb955-multus-daemon-config\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966528 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-var-lib-kubelet\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-os-release\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-run\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.977665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlvf\" (UniqueName: \"kubernetes.io/projected/3af9ad08-fbcf-4275-97c8-d7c990f050e3-kube-api-access-cnlvf\") pod \"iptables-alerter-5lnr9\" (UID: \"3af9ad08-fbcf-4275-97c8-d7c990f050e3\") " pod="openshift-network-operator/iptables-alerter-5lnr9" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d92505bf-3059-4df8-8286-9cbd3bf693c1-os-release\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ttg9\" (UniqueName: \"kubernetes.io/projected/d92505bf-3059-4df8-8286-9cbd3bf693c1-kube-api-access-8ttg9\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966722 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-systemd-units\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/301f4b24-c1c9-4754-ae99-e606ac261c2e-hosts-file\") pod \"node-resolver-8lgmt\" (UID: \"301f4b24-c1c9-4754-ae99-e606ac261c2e\") " pod="openshift-dns/node-resolver-8lgmt" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-run-openvswitch\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966805 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27hv\" (UniqueName: \"kubernetes.io/projected/ed8e9626-cece-4f7e-a51a-8a6a82082f09-kube-api-access-d27hv\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vs4\" (UniqueName: \"kubernetes.io/projected/f1261605-62c9-473d-bf74-156b31afd9c6-kube-api-access-x2vs4\") pod \"node-ca-h6xgq\" (UID: \"f1261605-62c9-473d-bf74-156b31afd9c6\") " pod="openshift-image-registry/node-ca-h6xgq" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c13136c-d5dd-438f-831e-367fcff88d3a-konnectivity-ca\") pod \"konnectivity-agent-mw2d4\" (UID: \"1c13136c-d5dd-438f-831e-367fcff88d3a\") " pod="kube-system/konnectivity-agent-mw2d4" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/301f4b24-c1c9-4754-ae99-e606ac261c2e-tmp-dir\") pod \"node-resolver-8lgmt\" (UID: \"301f4b24-c1c9-4754-ae99-e606ac261c2e\") " pod="openshift-dns/node-resolver-8lgmt" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966918 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-multus-socket-dir-parent\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966943 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-host\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.966947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-multus-conf-dir\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967038 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-ovnkube-config\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967119 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-var-lib-kubelet\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967135 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-sys\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:45.979459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-run-multus-certs\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-etc-openvswitch\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af9ad08-fbcf-4275-97c8-d7c990f050e3-iptables-alerter-script\") pod \"iptables-alerter-5lnr9\" (UID: \"3af9ad08-fbcf-4275-97c8-d7c990f050e3\") " pod="openshift-network-operator/iptables-alerter-5lnr9" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967526 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-run\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967700 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af9ad08-fbcf-4275-97c8-d7c990f050e3-host-slash\") pod \"iptables-alerter-5lnr9\" (UID: \"3af9ad08-fbcf-4275-97c8-d7c990f050e3\") " pod="openshift-network-operator/iptables-alerter-5lnr9" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967528 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af9ad08-fbcf-4275-97c8-d7c990f050e3-host-slash\") pod \"iptables-alerter-5lnr9\" (UID: \"3af9ad08-fbcf-4275-97c8-d7c990f050e3\") " pod="openshift-network-operator/iptables-alerter-5lnr9" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-hostroot\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-log-socket\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-cni-bin\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-ovn-node-metrics-cert\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.967970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnkk\" (UniqueName: \"kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk\") pod \"network-check-target-xnrdp\" (UID: \"84eade1e-4ce4-4c40-84f7-85ea4dfae29b\") " pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968042 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdc6\" (UniqueName: \"kubernetes.io/projected/6acafeb4-4bd5-4969-8722-6003504206b0-kube-api-access-6zdc6\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-kubelet-config\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d92505bf-3059-4df8-8286-9cbd3bf693c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af9ad08-fbcf-4275-97c8-d7c990f050e3-iptables-alerter-script\") pod \"iptables-alerter-5lnr9\" (UID: \"3af9ad08-fbcf-4275-97c8-d7c990f050e3\") " pod="openshift-network-operator/iptables-alerter-5lnr9" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-var-lib-openvswitch\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.981553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t44lr\" (UniqueName: \"kubernetes.io/projected/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-kube-api-access-t44lr\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mlk\" (UniqueName: \"kubernetes.io/projected/0cee993f-a067-44f3-8e7d-94a3e2529903-kube-api-access-48mlk\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-systemd\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d92505bf-3059-4df8-8286-9cbd3bf693c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-registration-dir\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-etc-selinux\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-sys-fs\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968477 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-systemd\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-sysconfig\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-kubernetes\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968597 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-kubernetes\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968605 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-sysconfig\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-sysctl-conf\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-lib-modules\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968722 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxv4\" (UniqueName: \"kubernetes.io/projected/301f4b24-c1c9-4754-ae99-e606ac261c2e-kube-api-access-jmxv4\") pod \"node-resolver-8lgmt\" (UID: \"301f4b24-c1c9-4754-ae99-e606ac261c2e\") " pod="openshift-dns/node-resolver-8lgmt" Jul 24 00:13:45.982479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968746 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-var-lib-cni-multus\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968772 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d92505bf-3059-4df8-8286-9cbd3bf693c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-multus-cni-dir\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-run-netns\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.968904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghpbj\" (UniqueName: \"kubernetes.io/projected/e6ffc7b0-5f19-4258-947c-95b01d7cb955-kube-api-access-ghpbj\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969028 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-lib-modules\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-run-netns\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969044 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-sysctl-conf\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-ovnkube-script-lib\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-socket-dir\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed8e9626-cece-4f7e-a51a-8a6a82082f09-tmp\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1261605-62c9-473d-bf74-156b31afd9c6-serviceca\") pod \"node-ca-h6xgq\" (UID: \"f1261605-62c9-473d-bf74-156b31afd9c6\") " pod="openshift-image-registry/node-ca-h6xgq" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-var-lib-kubelet\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-tuned\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1261605-62c9-473d-bf74-156b31afd9c6-host\") pod \"node-ca-h6xgq\" (UID: \"f1261605-62c9-473d-bf74-156b31afd9c6\") " pod="openshift-image-registry/node-ca-h6xgq" Jul 24 00:13:45.985914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-dbus\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-system-cni-dir\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-cni-netd\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-device-dir\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-cnibin\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-slash\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-run-ovn\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-env-overrides\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-modprobe-d\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1261605-62c9-473d-bf74-156b31afd9c6-host\") pod \"node-ca-h6xgq\" (UID: \"f1261605-62c9-473d-bf74-156b31afd9c6\") " pod="openshift-image-registry/node-ca-h6xgq" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-sysctl-d\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c13136c-d5dd-438f-831e-367fcff88d3a-agent-certs\") pod \"konnectivity-agent-mw2d4\" (UID: \"1c13136c-d5dd-438f-831e-367fcff88d3a\") " pod="kube-system/konnectivity-agent-mw2d4" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969604 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d92505bf-3059-4df8-8286-9cbd3bf693c1-system-cni-dir\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969730 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1261605-62c9-473d-bf74-156b31afd9c6-serviceca\") pod \"node-ca-h6xgq\" (UID: \"f1261605-62c9-473d-bf74-156b31afd9c6\") " pod="openshift-image-registry/node-ca-h6xgq" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969737 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-sysctl-d\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.969736 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-modprobe-d\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.973490 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed8e9626-cece-4f7e-a51a-8a6a82082f09-etc-tuned\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:45.986767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:45.973641 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed8e9626-cece-4f7e-a51a-8a6a82082f09-tmp\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:46.000133 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:45.999602 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:13:46.000133 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:45.999630 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:13:46.000133 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:45.999646 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zdnkk for pod openshift-network-diagnostics/network-check-target-xnrdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:13:46.000133 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:45.999732 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk podName:84eade1e-4ce4-4c40-84f7-85ea4dfae29b nodeName:}" failed. No retries permitted until 2026-07-24 00:13:46.499705846 +0000 UTC m=+3.072802638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zdnkk" (UniqueName: "kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk") pod "network-check-target-xnrdp" (UID: "84eade1e-4ce4-4c40-84f7-85ea4dfae29b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.003914 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnlvf\" (UniqueName: \"kubernetes.io/projected/3af9ad08-fbcf-4275-97c8-d7c990f050e3-kube-api-access-cnlvf\") pod \"iptables-alerter-5lnr9\" (UID: \"3af9ad08-fbcf-4275-97c8-d7c990f050e3\") " pod="openshift-network-operator/iptables-alerter-5lnr9" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.004402 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27hv\" (UniqueName: \"kubernetes.io/projected/ed8e9626-cece-4f7e-a51a-8a6a82082f09-kube-api-access-d27hv\") pod \"tuned-xxscj\" (UID: \"ed8e9626-cece-4f7e-a51a-8a6a82082f09\") " pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.004427 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vs4\" (UniqueName: \"kubernetes.io/projected/f1261605-62c9-473d-bf74-156b31afd9c6-kube-api-access-x2vs4\") pod \"node-ca-h6xgq\" (UID: \"f1261605-62c9-473d-bf74-156b31afd9c6\") " pod="openshift-image-registry/node-ca-h6xgq" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.012594 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-162.ec2.internal" event={"ID":"facb852d64cc5d31f3a76f06db558e9e","Type":"ContainerStarted","Data":"3bfb49c19762aace4b2d4deab6a2dbe8a97d0d18c430218f868813e520c49e74"} Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.013429 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" event={"ID":"893e9e357457367d0e01aa3fbac60606","Type":"ContainerStarted","Data":"f640553b3e4f9eab474e9753b354e62047780321b91b5783e9154e4ff08f893d"} Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-cni-bin\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-ovn-node-metrics-cert\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdc6\" (UniqueName: \"kubernetes.io/projected/6acafeb4-4bd5-4969-8722-6003504206b0-kube-api-access-6zdc6\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-kubelet-config\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d92505bf-3059-4df8-8286-9cbd3bf693c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-var-lib-openvswitch\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t44lr\" (UniqueName: \"kubernetes.io/projected/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-kube-api-access-t44lr\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mlk\" (UniqueName: \"kubernetes.io/projected/0cee993f-a067-44f3-8e7d-94a3e2529903-kube-api-access-48mlk\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d92505bf-3059-4df8-8286-9cbd3bf693c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-registration-dir\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-etc-selinux\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.090944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-sys-fs\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxv4\" (UniqueName: \"kubernetes.io/projected/301f4b24-c1c9-4754-ae99-e606ac261c2e-kube-api-access-jmxv4\") pod \"node-resolver-8lgmt\" (UID: \"301f4b24-c1c9-4754-ae99-e606ac261c2e\") " pod="openshift-dns/node-resolver-8lgmt" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-var-lib-cni-multus\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d92505bf-3059-4df8-8286-9cbd3bf693c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-multus-cni-dir\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-run-netns\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghpbj\" (UniqueName: \"kubernetes.io/projected/e6ffc7b0-5f19-4258-947c-95b01d7cb955-kube-api-access-ghpbj\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-run-netns\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-ovnkube-script-lib\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-socket-dir\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-var-lib-kubelet\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-dbus\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-system-cni-dir\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-cni-netd\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.094357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-device-dir\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.070996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-cnibin\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-slash\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-run-ovn\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-env-overrides\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c13136c-d5dd-438f-831e-367fcff88d3a-agent-certs\") pod \"konnectivity-agent-mw2d4\" (UID: \"1c13136c-d5dd-438f-831e-367fcff88d3a\") " pod="kube-system/konnectivity-agent-mw2d4" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d92505bf-3059-4df8-8286-9cbd3bf693c1-system-cni-dir\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-run-k8s-cni-cncf-io\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-etc-kubernetes\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6ffc7b0-5f19-4258-947c-95b01d7cb955-cni-binary-copy\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-kubelet\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-node-log\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d92505bf-3059-4df8-8286-9cbd3bf693c1-cnibin\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d92505bf-3059-4df8-8286-9cbd3bf693c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-var-lib-cni-bin\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-run-systemd\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071428 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e6ffc7b0-5f19-4258-947c-95b01d7cb955-multus-daemon-config\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-os-release\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.095466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d92505bf-3059-4df8-8286-9cbd3bf693c1-os-release\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ttg9\" (UniqueName: \"kubernetes.io/projected/d92505bf-3059-4df8-8286-9cbd3bf693c1-kube-api-access-8ttg9\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-systemd-units\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/301f4b24-c1c9-4754-ae99-e606ac261c2e-hosts-file\") pod \"node-resolver-8lgmt\" (UID: \"301f4b24-c1c9-4754-ae99-e606ac261c2e\") " pod="openshift-dns/node-resolver-8lgmt" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-run-openvswitch\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c13136c-d5dd-438f-831e-367fcff88d3a-konnectivity-ca\") pod \"konnectivity-agent-mw2d4\" (UID: \"1c13136c-d5dd-438f-831e-367fcff88d3a\") " pod="kube-system/konnectivity-agent-mw2d4" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/301f4b24-c1c9-4754-ae99-e606ac261c2e-tmp-dir\") pod \"node-resolver-8lgmt\" (UID: \"301f4b24-c1c9-4754-ae99-e606ac261c2e\") " pod="openshift-dns/node-resolver-8lgmt" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-multus-socket-dir-parent\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-multus-conf-dir\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-ovnkube-config\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-run-multus-certs\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-etc-openvswitch\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-hostroot\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-log-socket\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071926 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-var-lib-cni-multus\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.098718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.071998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-log-socket\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-etc-selinux\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072035 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-sys-fs\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-slash\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072128 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-registration-dir\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.072180 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072227 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072227 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-kubelet-config\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.072260 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret podName:3c21f04e-dbb0-4743-831f-462ea9ffcd9c nodeName:}" failed. No retries permitted until 2026-07-24 00:13:46.572239053 +0000 UTC m=+3.145335834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret") pod "global-pull-secret-syncer-vncds" (UID: "3c21f04e-dbb0-4743-831f-462ea9ffcd9c") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072386 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-multus-cni-dir\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072440 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-run-netns\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072570 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d92505bf-3059-4df8-8286-9cbd3bf693c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072629 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-run-netns\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072675 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-dbus\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072710 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d92505bf-3059-4df8-8286-9cbd3bf693c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072749 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-env-overrides\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d92505bf-3059-4df8-8286-9cbd3bf693c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.099634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072835 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-run-ovn\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072878 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-multus-conf-dir\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072882 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-var-lib-openvswitch\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072945 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-systemd-units\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072969 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-etc-openvswitch\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/301f4b24-c1c9-4754-ae99-e606ac261c2e-hosts-file\") pod \"node-resolver-8lgmt\" (UID: \"301f4b24-c1c9-4754-ae99-e606ac261c2e\") " pod="openshift-dns/node-resolver-8lgmt" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073065 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-run-openvswitch\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073067 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d92505bf-3059-4df8-8286-9cbd3bf693c1-system-cni-dir\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.072182 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-cni-bin\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-etc-kubernetes\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073169 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-run-k8s-cni-cncf-io\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-hostroot\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073235 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-var-lib-cni-bin\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073262 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-ovnkube-script-lib\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073285 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-kubelet\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073325 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-node-log\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073362 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-os-release\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.103299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d92505bf-3059-4df8-8286-9cbd3bf693c1-cnibin\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-socket-dir\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-run-systemd\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073412 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-var-lib-kubelet\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073460 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073509 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-host-cni-netd\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073560 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-system-cni-dir\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073616 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c13136c-d5dd-438f-831e-367fcff88d3a-konnectivity-ca\") pod \"konnectivity-agent-mw2d4\" (UID: \"1c13136c-d5dd-438f-831e-367fcff88d3a\") " pod="kube-system/konnectivity-agent-mw2d4" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073648 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0cee993f-a067-44f3-8e7d-94a3e2529903-device-dir\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073702 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-multus-socket-dir-parent\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073750 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6ffc7b0-5f19-4258-947c-95b01d7cb955-cni-binary-copy\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.073778 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.073828 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs podName:6acafeb4-4bd5-4969-8722-6003504206b0 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:46.573810999 +0000 UTC m=+3.146907782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs") pod "network-metrics-daemon-9dmnl" (UID: "6acafeb4-4bd5-4969-8722-6003504206b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073836 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d92505bf-3059-4df8-8286-9cbd3bf693c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d92505bf-3059-4df8-8286-9cbd3bf693c1-os-release\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073880 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-host-run-multus-certs\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073882 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e6ffc7b0-5f19-4258-947c-95b01d7cb955-multus-daemon-config\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.103950 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073919 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6ffc7b0-5f19-4258-947c-95b01d7cb955-cnibin\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.104606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.073921 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/301f4b24-c1c9-4754-ae99-e606ac261c2e-tmp-dir\") pod \"node-resolver-8lgmt\" (UID: \"301f4b24-c1c9-4754-ae99-e606ac261c2e\") " pod="openshift-dns/node-resolver-8lgmt" Jul 24 00:13:46.104606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.074087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-ovnkube-config\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.104606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.075123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-ovn-node-metrics-cert\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.104606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.076210 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c13136c-d5dd-438f-831e-367fcff88d3a-agent-certs\") pod \"konnectivity-agent-mw2d4\" (UID: \"1c13136c-d5dd-438f-831e-367fcff88d3a\") " pod="kube-system/konnectivity-agent-mw2d4" Jul 24 00:13:46.104606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.086479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mlk\" (UniqueName: \"kubernetes.io/projected/0cee993f-a067-44f3-8e7d-94a3e2529903-kube-api-access-48mlk\") pod \"aws-ebs-csi-driver-node-m776p\" (UID: \"0cee993f-a067-44f3-8e7d-94a3e2529903\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.104606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.086839 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghpbj\" (UniqueName: \"kubernetes.io/projected/e6ffc7b0-5f19-4258-947c-95b01d7cb955-kube-api-access-ghpbj\") pod \"multus-jzwjn\" (UID: \"e6ffc7b0-5f19-4258-947c-95b01d7cb955\") " pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.104606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.087139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdc6\" (UniqueName: \"kubernetes.io/projected/6acafeb4-4bd5-4969-8722-6003504206b0-kube-api-access-6zdc6\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:46.104606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.087937 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t44lr\" (UniqueName: \"kubernetes.io/projected/bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9-kube-api-access-t44lr\") pod \"ovnkube-node-dtgbd\" (UID: \"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.104606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.089387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxv4\" (UniqueName: \"kubernetes.io/projected/301f4b24-c1c9-4754-ae99-e606ac261c2e-kube-api-access-jmxv4\") pod \"node-resolver-8lgmt\" (UID: \"301f4b24-c1c9-4754-ae99-e606ac261c2e\") " pod="openshift-dns/node-resolver-8lgmt" Jul 24 00:13:46.104606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.089466 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ttg9\" (UniqueName: \"kubernetes.io/projected/d92505bf-3059-4df8-8286-9cbd3bf693c1-kube-api-access-8ttg9\") pod \"multus-additional-cni-plugins-vt9rk\" (UID: \"d92505bf-3059-4df8-8286-9cbd3bf693c1\") " pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.201909 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.160107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xxscj" Jul 24 00:13:46.201909 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.173344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h6xgq" Jul 24 00:13:46.201909 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.188235 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 00:13:46.288931 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.222871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5lnr9" Jul 24 00:13:46.288931 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.236097 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:13:46.288931 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:46.236988 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af9ad08_fbcf_4275_97c8_d7c990f050e3.slice/crio-bde7484a5a13edfc305e0feb4aa2fdba2fedb3af52bcd7b0d6a677b9683dcad0 WatchSource:0}: Error finding container bde7484a5a13edfc305e0feb4aa2fdba2fedb3af52bcd7b0d6a677b9683dcad0: Status 404 returned error can't find the container with id bde7484a5a13edfc305e0feb4aa2fdba2fedb3af52bcd7b0d6a677b9683dcad0 Jul 24 00:13:46.288931 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:46.244964 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf497b44_5da7_4ff0_91c2_d7ea5a9db4a9.slice/crio-c13d4ee20480f75ed50388a464e83aa9557ab386ac0e6e9cfe1b07d153ea3606 WatchSource:0}: Error finding container c13d4ee20480f75ed50388a464e83aa9557ab386ac0e6e9cfe1b07d153ea3606: Status 404 returned error can't find the container with id c13d4ee20480f75ed50388a464e83aa9557ab386ac0e6e9cfe1b07d153ea3606 Jul 24 00:13:46.288931 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.245683 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mw2d4" Jul 24 00:13:46.288931 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.252759 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" Jul 24 00:13:46.288931 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.263278 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8lgmt" Jul 24 00:13:46.288931 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.267242 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 00:13:46.302344 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.301872 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vt9rk" Jul 24 00:13:46.315053 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.314171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jzwjn" Jul 24 00:13:46.329009 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:46.328982 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c13136c_d5dd_438f_831e_367fcff88d3a.slice/crio-b276a358928dab4386904029abfe71ba2139bf19e9d1fd939b6631310e46d4dd WatchSource:0}: Error finding container b276a358928dab4386904029abfe71ba2139bf19e9d1fd939b6631310e46d4dd: Status 404 returned error can't find the container with id b276a358928dab4386904029abfe71ba2139bf19e9d1fd939b6631310e46d4dd Jul 24 00:13:46.332679 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:46.332614 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301f4b24_c1c9_4754_ae99_e606ac261c2e.slice/crio-ec1e227faf4494df2b6c2c11913bcb3af928a9c1c26f6b728ade1b0669c5a586 WatchSource:0}: Error finding container ec1e227faf4494df2b6c2c11913bcb3af928a9c1c26f6b728ade1b0669c5a586: Status 404 returned error can't find the container with id ec1e227faf4494df2b6c2c11913bcb3af928a9c1c26f6b728ade1b0669c5a586 Jul 24 00:13:46.335205 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:13:46.335158 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ffc7b0_5f19_4258_947c_95b01d7cb955.slice/crio-47a8e12223b26bff13421903faee5786b88358dd74690c54122a88bc066c562b WatchSource:0}: Error finding container 47a8e12223b26bff13421903faee5786b88358dd74690c54122a88bc066c562b: Status 404 returned error can't find the container with id 47a8e12223b26bff13421903faee5786b88358dd74690c54122a88bc066c562b Jul 24 00:13:46.575575 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.575476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnkk\" (UniqueName: \"kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk\") pod \"network-check-target-xnrdp\" (UID: \"84eade1e-4ce4-4c40-84f7-85ea4dfae29b\") " pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:46.575575 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.575533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:46.575823 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.575592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:46.575823 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.575671 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:13:46.575823 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.575684 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:13:46.575823 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.575703 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:13:46.575823 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.575718 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zdnkk for pod openshift-network-diagnostics/network-check-target-xnrdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:13:46.575823 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.575702 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:13:46.575823 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.575753 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret podName:3c21f04e-dbb0-4743-831f-462ea9ffcd9c nodeName:}" failed. No retries permitted until 2026-07-24 00:13:47.575733043 +0000 UTC m=+4.148829827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret") pod "global-pull-secret-syncer-vncds" (UID: "3c21f04e-dbb0-4743-831f-462ea9ffcd9c") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:13:46.575823 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.575773 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs podName:6acafeb4-4bd5-4969-8722-6003504206b0 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:47.575762734 +0000 UTC m=+4.148859525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs") pod "network-metrics-daemon-9dmnl" (UID: "6acafeb4-4bd5-4969-8722-6003504206b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:13:46.575823 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.575787 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk podName:84eade1e-4ce4-4c40-84f7-85ea4dfae29b nodeName:}" failed. No retries permitted until 2026-07-24 00:13:47.575779965 +0000 UTC m=+4.148876763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zdnkk" (UniqueName: "kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk") pod "network-check-target-xnrdp" (UID: "84eade1e-4ce4-4c40-84f7-85ea4dfae29b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:13:46.899359 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.899215 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 00:08:46 +0000 UTC" deadline="2028-02-18 14:04:18.46421723 +0000 UTC" Jul 24 00:13:46.899359 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.899301 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13789h50m31.564921661s" Jul 24 00:13:46.996502 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.996468 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:46.997132 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:46.996508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:46.997132 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.996622 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:13:46.997132 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:46.996717 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:13:47.017058 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.016998 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" event={"ID":"0cee993f-a067-44f3-8e7d-94a3e2529903","Type":"ContainerStarted","Data":"44a8013ab5c68f538782e9b002500a87aab1460e832f32ba97efd82307ca7096"} Jul 24 00:13:47.018648 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.018589 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jzwjn" event={"ID":"e6ffc7b0-5f19-4258-947c-95b01d7cb955","Type":"ContainerStarted","Data":"47a8e12223b26bff13421903faee5786b88358dd74690c54122a88bc066c562b"} Jul 24 00:13:47.019894 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.019858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mw2d4" event={"ID":"1c13136c-d5dd-438f-831e-367fcff88d3a","Type":"ContainerStarted","Data":"b276a358928dab4386904029abfe71ba2139bf19e9d1fd939b6631310e46d4dd"} Jul 24 00:13:47.021333 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.021292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" event={"ID":"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9","Type":"ContainerStarted","Data":"c13d4ee20480f75ed50388a464e83aa9557ab386ac0e6e9cfe1b07d153ea3606"} Jul 24 00:13:47.022700 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.022675 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5lnr9" event={"ID":"3af9ad08-fbcf-4275-97c8-d7c990f050e3","Type":"ContainerStarted","Data":"bde7484a5a13edfc305e0feb4aa2fdba2fedb3af52bcd7b0d6a677b9683dcad0"} Jul 24 00:13:47.024142 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.024118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vt9rk" event={"ID":"d92505bf-3059-4df8-8286-9cbd3bf693c1","Type":"ContainerStarted","Data":"babcdebcc6abe56f3437c41c37a5d350b8b746ecc1c0bdcb2d155c76ce9df29e"} Jul 24 00:13:47.025539 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.025509 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8lgmt" event={"ID":"301f4b24-c1c9-4754-ae99-e606ac261c2e","Type":"ContainerStarted","Data":"ec1e227faf4494df2b6c2c11913bcb3af928a9c1c26f6b728ade1b0669c5a586"} Jul 24 00:13:47.027157 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.027131 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h6xgq" event={"ID":"f1261605-62c9-473d-bf74-156b31afd9c6","Type":"ContainerStarted","Data":"6e1a3b00f190d8afa32ebf376da0f81a8c07394481ce528e148fb10e6925b03c"} Jul 24 00:13:47.028574 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.028516 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xxscj" event={"ID":"ed8e9626-cece-4f7e-a51a-8a6a82082f09","Type":"ContainerStarted","Data":"e6ebfe28ca581fb91482a3bda8a711b2dbf9d9c4bbed25412e36bcc37e31b152"} Jul 24 00:13:47.585496 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.585453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnkk\" (UniqueName: \"kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk\") pod \"network-check-target-xnrdp\" (UID: \"84eade1e-4ce4-4c40-84f7-85ea4dfae29b\") " pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:47.585713 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.585523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:47.585713 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.585587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:47.585713 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:47.585622 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:13:47.585713 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:47.585650 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:13:47.585713 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:47.585665 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zdnkk for pod openshift-network-diagnostics/network-check-target-xnrdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:13:47.585713 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:47.585678 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:13:47.586134 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:47.585678 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:13:47.586134 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:47.585732 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk podName:84eade1e-4ce4-4c40-84f7-85ea4dfae29b nodeName:}" failed. No retries permitted until 2026-07-24 00:13:49.585708698 +0000 UTC m=+6.158805497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zdnkk" (UniqueName: "kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk") pod "network-check-target-xnrdp" (UID: "84eade1e-4ce4-4c40-84f7-85ea4dfae29b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:13:47.586134 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:47.585762 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs podName:6acafeb4-4bd5-4969-8722-6003504206b0 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:49.585743784 +0000 UTC m=+6.158840583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs") pod "network-metrics-daemon-9dmnl" (UID: "6acafeb4-4bd5-4969-8722-6003504206b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:13:47.586134 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:47.585780 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret podName:3c21f04e-dbb0-4743-831f-462ea9ffcd9c nodeName:}" failed. No retries permitted until 2026-07-24 00:13:49.585770204 +0000 UTC m=+6.158866989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret") pod "global-pull-secret-syncer-vncds" (UID: "3c21f04e-dbb0-4743-831f-462ea9ffcd9c") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:13:47.900083 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.899830 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 00:08:46 +0000 UTC" deadline="2028-01-29 16:08:21.36206536 +0000 UTC" Jul 24 00:13:47.900083 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.899876 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13311h54m33.46219589s" Jul 24 00:13:47.996547 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:47.996286 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:47.997137 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:47.996663 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:13:48.042043 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:48.039269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-162.ec2.internal" event={"ID":"facb852d64cc5d31f3a76f06db558e9e","Type":"ContainerStarted","Data":"52e23791fad384e7db1f9897f8cfa01260b65b04cea74a2dd52a42bf2305f425"} Jul 24 00:13:48.056616 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:48.056551 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-162.ec2.internal" podStartSLOduration=3.056527562 podStartE2EDuration="3.056527562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:13:48.055304398 +0000 UTC m=+4.628401204" watchObservedRunningTime="2026-07-24 00:13:48.056527562 +0000 UTC m=+4.629624370" Jul 24 00:13:48.995866 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:48.995831 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:48.996152 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:48.996074 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:13:48.996514 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:48.996475 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:48.996795 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:48.996591 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:13:49.053241 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:49.053202 2576 generic.go:356] "Generic (PLEG): container finished" podID="893e9e357457367d0e01aa3fbac60606" containerID="f099ff9e485345abbbc474ee02ddb110267dd87db56cf2aee5e5d972dc03b2a4" exitCode=0 Jul 24 00:13:49.054087 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:49.054063 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" event={"ID":"893e9e357457367d0e01aa3fbac60606","Type":"ContainerDied","Data":"f099ff9e485345abbbc474ee02ddb110267dd87db56cf2aee5e5d972dc03b2a4"} Jul 24 00:13:49.607273 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:49.607228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:49.607561 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:49.607291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnkk\" (UniqueName: \"kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk\") pod \"network-check-target-xnrdp\" (UID: \"84eade1e-4ce4-4c40-84f7-85ea4dfae29b\") " pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:49.607561 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:49.607334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:49.607561 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:49.607500 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:13:49.607760 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:49.607572 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret podName:3c21f04e-dbb0-4743-831f-462ea9ffcd9c nodeName:}" failed. No retries permitted until 2026-07-24 00:13:53.607552959 +0000 UTC m=+10.180649740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret") pod "global-pull-secret-syncer-vncds" (UID: "3c21f04e-dbb0-4743-831f-462ea9ffcd9c") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:13:49.608020 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:49.608002 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:13:49.608093 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:49.608059 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs podName:6acafeb4-4bd5-4969-8722-6003504206b0 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:53.608041232 +0000 UTC m=+10.181138018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs") pod "network-metrics-daemon-9dmnl" (UID: "6acafeb4-4bd5-4969-8722-6003504206b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:13:49.608168 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:49.608139 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:13:49.608168 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:49.608153 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:13:49.608168 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:49.608165 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zdnkk for pod openshift-network-diagnostics/network-check-target-xnrdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:13:49.608317 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:49.608200 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk podName:84eade1e-4ce4-4c40-84f7-85ea4dfae29b nodeName:}" failed. No retries permitted until 2026-07-24 00:13:53.608189309 +0000 UTC m=+10.181286094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zdnkk" (UniqueName: "kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk") pod "network-check-target-xnrdp" (UID: "84eade1e-4ce4-4c40-84f7-85ea4dfae29b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:13:49.997659 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:49.997148 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:49.997659 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:49.997277 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:13:50.996181 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:50.995667 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:50.996181 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:50.995794 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:13:50.996181 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:50.995913 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:50.996181 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:50.996047 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:13:51.998181 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:51.997710 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:51.998181 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:51.997836 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:13:52.995422 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:52.995382 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:52.995620 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:52.995511 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:13:52.995971 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:52.995920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:52.996099 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:52.996063 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:13:53.644014 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:53.643980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnkk\" (UniqueName: \"kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk\") pod \"network-check-target-xnrdp\" (UID: \"84eade1e-4ce4-4c40-84f7-85ea4dfae29b\") " pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:53.644533 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:53.644030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:53.644533 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:53.644069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:53.644533 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:53.644165 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:13:53.644533 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:53.644223 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs podName:6acafeb4-4bd5-4969-8722-6003504206b0 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:01.644210395 +0000 UTC m=+18.217307182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs") pod "network-metrics-daemon-9dmnl" (UID: "6acafeb4-4bd5-4969-8722-6003504206b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:13:53.644842 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:53.644560 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:13:53.644842 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:53.644573 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:13:53.644842 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:53.644583 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zdnkk for pod openshift-network-diagnostics/network-check-target-xnrdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:13:53.644842 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:53.644623 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk podName:84eade1e-4ce4-4c40-84f7-85ea4dfae29b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:01.644607832 +0000 UTC m=+18.217704616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zdnkk" (UniqueName: "kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk") pod "network-check-target-xnrdp" (UID: "84eade1e-4ce4-4c40-84f7-85ea4dfae29b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:13:53.644842 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:53.644666 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:13:53.644842 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:53.644689 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret podName:3c21f04e-dbb0-4743-831f-462ea9ffcd9c nodeName:}" failed. No retries permitted until 2026-07-24 00:14:01.644681797 +0000 UTC m=+18.217778577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret") pod "global-pull-secret-syncer-vncds" (UID: "3c21f04e-dbb0-4743-831f-462ea9ffcd9c") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:13:53.997337 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:53.995712 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:53.997337 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:53.995846 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:13:54.995447 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:54.995404 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:54.995447 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:54.995446 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:54.995999 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:54.995592 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:13:54.995999 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:54.995685 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:13:55.995599 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:55.995565 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:55.996043 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:55.995702 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:13:56.995994 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:56.995936 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:56.995994 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:56.995974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:56.996427 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:56.996094 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:13:56.996427 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:56.996249 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:13:57.995802 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:57.995765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:57.996009 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:57.995893 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:13:58.995745 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:58.995712 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:13:58.996225 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:58.995759 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:13:58.996225 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:58.995854 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:13:58.996225 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:58.995993 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:13:59.995818 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:13:59.995784 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:13:59.996261 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:13:59.995913 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:14:00.996128 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:00.996089 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:00.996654 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:00.996228 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:14:00.996654 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:00.996278 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:00.996654 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:00.996396 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:14:01.705132 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:01.705096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:01.705298 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:01.705143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnkk\" (UniqueName: \"kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk\") pod \"network-check-target-xnrdp\" (UID: \"84eade1e-4ce4-4c40-84f7-85ea4dfae29b\") " pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:01.705298 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:01.705166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:01.705298 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:01.705267 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:14:01.705298 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:01.705275 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:14:01.705474 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:01.705284 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:14:01.705474 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:01.705328 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret podName:3c21f04e-dbb0-4743-831f-462ea9ffcd9c nodeName:}" failed. No retries permitted until 2026-07-24 00:14:17.705308935 +0000 UTC m=+34.278405739 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret") pod "global-pull-secret-syncer-vncds" (UID: "3c21f04e-dbb0-4743-831f-462ea9ffcd9c") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:14:01.705474 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:01.705346 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs podName:6acafeb4-4bd5-4969-8722-6003504206b0 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:17.705338738 +0000 UTC m=+34.278435519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs") pod "network-metrics-daemon-9dmnl" (UID: "6acafeb4-4bd5-4969-8722-6003504206b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:14:01.705474 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:01.705346 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:14:01.705474 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:01.705363 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zdnkk for pod openshift-network-diagnostics/network-check-target-xnrdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:14:01.705474 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:01.705409 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk podName:84eade1e-4ce4-4c40-84f7-85ea4dfae29b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:17.70539219 +0000 UTC m=+34.278488970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zdnkk" (UniqueName: "kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk") pod "network-check-target-xnrdp" (UID: "84eade1e-4ce4-4c40-84f7-85ea4dfae29b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:14:01.995570 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:01.995491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:01.995737 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:01.995611 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:14:02.995578 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:02.995538 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:02.996021 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:02.995542 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:02.996021 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:02.995655 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:14:02.996021 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:02.995777 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:14:03.996862 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:03.996826 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:03.997339 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:03.996940 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:14:04.996113 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:04.996082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:04.996249 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:04.996082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:04.996249 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:04.996177 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:14:04.996324 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:04.996261 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:14:05.996268 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:05.995938 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:05.997056 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:05.996380 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:14:06.088760 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.088722 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h6xgq" event={"ID":"f1261605-62c9-473d-bf74-156b31afd9c6","Type":"ContainerStarted","Data":"4da9c3a7368c259313a5882009e86ca11d3c03cd1b63d6f9d8f80190bbfa2282"} Jul 24 00:14:06.090331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.090303 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xxscj" event={"ID":"ed8e9626-cece-4f7e-a51a-8a6a82082f09","Type":"ContainerStarted","Data":"de7f0d396f40d4de8b7650f2ba3dff6bd423b3a0e5b8e14296c1ae23952777c9"} Jul 24 00:14:06.093002 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.092970 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" event={"ID":"893e9e357457367d0e01aa3fbac60606","Type":"ContainerStarted","Data":"729bca757a7dd2cf803ae33da7a3eb0a7079c9ac777a16f6971107b4bec34eef"} Jul 24 00:14:06.094542 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.094477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" event={"ID":"0cee993f-a067-44f3-8e7d-94a3e2529903","Type":"ContainerStarted","Data":"f95b55f2f354de34a6ab70f6034c32e888c762f266697aacea6d29bf7a462ccb"} Jul 24 00:14:06.096008 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.095986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jzwjn" event={"ID":"e6ffc7b0-5f19-4258-947c-95b01d7cb955","Type":"ContainerStarted","Data":"166ac8d76790140a34eda12c70c78b49b6268c6ceb56c78c9216420fbb2e1734"} Jul 24 00:14:06.097628 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.097600 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mw2d4" event={"ID":"1c13136c-d5dd-438f-831e-367fcff88d3a","Type":"ContainerStarted","Data":"557090cf5787d11d438a0e287af20a5a129fa319f10f47a28cd262b15dbcb5f5"} Jul 24 00:14:06.101183 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.101128 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" event={"ID":"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9","Type":"ContainerStarted","Data":"0e91de5272797982c359f084abd0cb12877963b7edf261dd209fe6775250abf7"} Jul 24 00:14:06.101280 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.101199 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" event={"ID":"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9","Type":"ContainerStarted","Data":"4eb9bfbfb4040299e2624f3ff783be6d023b860b51f48a885362398888c602c5"} Jul 24 00:14:06.101280 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.101247 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" event={"ID":"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9","Type":"ContainerStarted","Data":"d0992b5c73c53c99531a1bf38cbf434ffc2edbe6e6e800b5b1c9b81039db602e"} Jul 24 00:14:06.101387 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.101294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" event={"ID":"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9","Type":"ContainerStarted","Data":"2d905e88e15358fbe3cfda23eefd4a2d2f968c1f03e892e31137cd7d4b76f3f9"} Jul 24 00:14:06.101387 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.101343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" event={"ID":"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9","Type":"ContainerStarted","Data":"5fca5ec948e23be01e3d264d38385b566ec3639f719507343262589cd83f2420"} Jul 24 00:14:06.101474 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.101391 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" event={"ID":"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9","Type":"ContainerStarted","Data":"152bc98e639bd2bce00a093338386d8a08b30cd07478026e153f10ef6dded58a"} Jul 24 00:14:06.102797 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.102735 2576 generic.go:356] "Generic (PLEG): container finished" podID="d92505bf-3059-4df8-8286-9cbd3bf693c1" containerID="2464414e3a2561acfbdae63d0c2e3f6fc2f6abff0b5dd0b489a0efbd42d67163" exitCode=0 Jul 24 00:14:06.102900 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.102815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vt9rk" event={"ID":"d92505bf-3059-4df8-8286-9cbd3bf693c1","Type":"ContainerDied","Data":"2464414e3a2561acfbdae63d0c2e3f6fc2f6abff0b5dd0b489a0efbd42d67163"} Jul 24 00:14:06.102900 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.102878 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h6xgq" podStartSLOduration=2.971418881 podStartE2EDuration="22.102859121s" totalImagesPullingTime="19.13144024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:06.102033516 +0000 UTC m=+22.675130319" watchObservedRunningTime="2026-07-24 00:14:06.102859121 +0000 UTC m=+22.675955924" Jul 24 00:14:06.105273 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.105238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8lgmt" event={"ID":"301f4b24-c1c9-4754-ae99-e606ac261c2e","Type":"ContainerStarted","Data":"f535b9c6909bb89eaea41ef4e42ffc124ae8b81143c1ea630b6883ebb3aad6f8"} Jul 24 00:14:06.115407 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.115364 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-162.ec2.internal" podStartSLOduration=21.115348422 podStartE2EDuration="21.115348422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:06.115280663 +0000 UTC m=+22.688377467" watchObservedRunningTime="2026-07-24 00:14:06.115348422 +0000 UTC m=+22.688445226" Jul 24 00:14:06.129935 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.129884 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xxscj" podStartSLOduration=2.915455058 podStartE2EDuration="22.129864415s" totalImagesPullingTime="19.214409357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:06.129779093 +0000 UTC m=+22.702875894" watchObservedRunningTime="2026-07-24 00:14:06.129864415 +0000 UTC m=+22.702961208" Jul 24 00:14:06.146102 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.146047 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jzwjn" podStartSLOduration=3.073794562 podStartE2EDuration="22.146026547s" totalImagesPullingTime="19.072231985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:06.145730909 +0000 UTC m=+22.718827723" watchObservedRunningTime="2026-07-24 00:14:06.146026547 +0000 UTC m=+22.719123350" Jul 24 00:14:06.159454 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.159403 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mw2d4" podStartSLOduration=3.135360796 podStartE2EDuration="22.159384579s" totalImagesPullingTime="19.024023783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:06.158857982 +0000 UTC m=+22.731954785" watchObservedRunningTime="2026-07-24 00:14:06.159384579 +0000 UTC m=+22.732481381" Jul 24 00:14:06.174629 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.174564 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8lgmt" podStartSLOduration=3.098588867 podStartE2EDuration="22.174542547s" totalImagesPullingTime="19.07595368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:06.173919263 +0000 UTC m=+22.747016066" watchObservedRunningTime="2026-07-24 00:14:06.174542547 +0000 UTC m=+22.747639351" Jul 24 00:14:06.819712 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.819618 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 00:14:06.964007 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.963832 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T00:14:06.819642596Z","UUID":"517e1927-3b9f-473b-a4b7-987c5b5bdb73","Handler":null,"Name":"","Endpoint":""} Jul 24 00:14:06.965353 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.965330 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 00:14:06.965466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.965371 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 00:14:06.996146 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.996121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:06.996286 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:06.996233 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:14:06.996286 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:06.996241 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:06.996746 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:06.996339 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:14:07.109923 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:07.109884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" event={"ID":"0cee993f-a067-44f3-8e7d-94a3e2529903","Type":"ContainerStarted","Data":"a893b03d18f6082236d0b89be45831f23e55ed943223337c18c15c530cc028be"} Jul 24 00:14:07.111353 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:07.111316 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5lnr9" event={"ID":"3af9ad08-fbcf-4275-97c8-d7c990f050e3","Type":"ContainerStarted","Data":"9a60f046a8307619366775a7eeb4e90dc2b59be5db11486caf6a32fbc9992bee"} Jul 24 00:14:07.126660 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:07.126606 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5lnr9" podStartSLOduration=3.9545443860000002 podStartE2EDuration="23.126588112s" totalImagesPullingTime="19.172043726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:07.126496861 +0000 UTC m=+23.699593662" watchObservedRunningTime="2026-07-24 00:14:07.126588112 +0000 UTC m=+23.699684915" Jul 24 00:14:07.995584 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:07.995555 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:07.995900 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:07.995647 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:14:08.996017 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:08.995698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:08.996458 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:08.995747 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:08.996458 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:08.996152 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:14:08.996458 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:08.996233 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:14:09.120157 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:09.120121 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" event={"ID":"0cee993f-a067-44f3-8e7d-94a3e2529903","Type":"ContainerStarted","Data":"1a1a443f3a3a13529baba416f17b02475f67af3df0840c3c3b7c8495f1ec7782"} Jul 24 00:14:09.123847 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:09.123824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" event={"ID":"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9","Type":"ContainerStarted","Data":"074d6cbc635e3ad243a52d4920d38e87bbf4fdbc79aaf000bb536bc9815f020c"} Jul 24 00:14:09.996247 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:09.996211 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:09.996824 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:09.996337 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:14:09.999452 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:09.999427 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mw2d4" Jul 24 00:14:09.999564 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:09.999510 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mw2d4" Jul 24 00:14:09.999774 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:09.999756 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mw2d4" Jul 24 00:14:10.000255 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:10.000239 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mw2d4" Jul 24 00:14:10.015603 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:10.015559 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m776p" podStartSLOduration=4.691567258 podStartE2EDuration="26.015544963s" totalImagesPullingTime="21.323977705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:09.137261768 +0000 UTC m=+25.710358570" watchObservedRunningTime="2026-07-24 00:14:10.015544963 +0000 UTC m=+26.588641766" Jul 24 00:14:10.995596 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:10.995434 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:10.995709 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:10.995469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:10.995709 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:10.995672 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:14:10.995790 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:10.995713 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:14:11.131130 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:11.131096 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" event={"ID":"bf497b44-5da7-4ff0-91c2-d7ea5a9db4a9","Type":"ContainerStarted","Data":"1fc6fa1109b42963aa7c60697d65a1454046eb0b3ec3abd7709df2ade6f6d4be"} Jul 24 00:14:11.131916 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:11.131370 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:14:11.131916 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:11.131432 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:14:11.131916 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:11.131492 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:14:11.133153 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:11.133124 2576 generic.go:356] "Generic (PLEG): container finished" podID="d92505bf-3059-4df8-8286-9cbd3bf693c1" containerID="f7867581262db973f0bcdb6eef67c42ae6068bc2290036fc2449de2566c5216f" exitCode=0 Jul 24 00:14:11.133289 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:11.133216 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vt9rk" event={"ID":"d92505bf-3059-4df8-8286-9cbd3bf693c1","Type":"ContainerDied","Data":"f7867581262db973f0bcdb6eef67c42ae6068bc2290036fc2449de2566c5216f"} Jul 24 00:14:11.146902 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:11.146852 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:14:11.147006 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:11.146940 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:14:11.156217 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:11.156181 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" podStartSLOduration=7.934660018 podStartE2EDuration="27.15616676s" totalImagesPullingTime="19.221506742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:11.154892006 +0000 UTC m=+27.727988808" watchObservedRunningTime="2026-07-24 00:14:11.15616676 +0000 UTC m=+27.729263562" Jul 24 00:14:11.996475 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:11.996440 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:11.996689 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:11.996580 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:14:12.678461 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:12.678382 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vncds"] Jul 24 00:14:12.678938 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:12.678475 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:12.678938 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:12.678569 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:14:12.681232 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:12.681191 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xnrdp"] Jul 24 00:14:12.681345 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:12.681295 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:12.681413 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:12.681380 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:14:12.681864 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:12.681840 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9dmnl"] Jul 24 00:14:12.681971 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:12.681936 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:12.682048 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:12.682029 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:14:13.139483 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:13.139263 2576 generic.go:356] "Generic (PLEG): container finished" podID="d92505bf-3059-4df8-8286-9cbd3bf693c1" containerID="844c8ff9e5718c34d36ea131d0a5b95f1e463a643f7371a30eac391a453c3760" exitCode=0 Jul 24 00:14:13.139483 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:13.139352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vt9rk" event={"ID":"d92505bf-3059-4df8-8286-9cbd3bf693c1","Type":"ContainerDied","Data":"844c8ff9e5718c34d36ea131d0a5b95f1e463a643f7371a30eac391a453c3760"} Jul 24 00:14:13.996283 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:13.996252 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:13.996736 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:13.996348 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:14:14.143304 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:14.143277 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vt9rk" event={"ID":"d92505bf-3059-4df8-8286-9cbd3bf693c1","Type":"ContainerStarted","Data":"046574e4a3aa3e4e3a43373b8cf78dcbf5e7cc32da3a22b2869de6f4c60fa53a"} Jul 24 00:14:14.996162 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:14.996080 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:14.996303 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:14.996092 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:14.996974 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:14.996556 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:14:14.996974 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:14.996725 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:14:15.147938 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:15.147898 2576 generic.go:356] "Generic (PLEG): container finished" podID="d92505bf-3059-4df8-8286-9cbd3bf693c1" containerID="046574e4a3aa3e4e3a43373b8cf78dcbf5e7cc32da3a22b2869de6f4c60fa53a" exitCode=0 Jul 24 00:14:15.148093 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:15.147963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vt9rk" event={"ID":"d92505bf-3059-4df8-8286-9cbd3bf693c1","Type":"ContainerDied","Data":"046574e4a3aa3e4e3a43373b8cf78dcbf5e7cc32da3a22b2869de6f4c60fa53a"} Jul 24 00:14:15.898803 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:15.898762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:15.981305 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:15.981252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:15.996064 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:15.996034 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:15.996218 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:15.996157 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:14:16.996210 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:16.996164 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:16.996210 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:16.996189 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:16.996685 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:16.996310 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:14:16.996685 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:16.996408 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:14:17.727781 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:17.727743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:17.728001 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:17.727792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnkk\" (UniqueName: \"kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk\") pod \"network-check-target-xnrdp\" (UID: \"84eade1e-4ce4-4c40-84f7-85ea4dfae29b\") " pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:17.728001 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:17.727911 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:14:17.728001 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:17.727930 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:17.728001 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:17.727916 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:14:17.728211 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:17.728011 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs podName:6acafeb4-4bd5-4969-8722-6003504206b0 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:49.72798725 +0000 UTC m=+66.301084036 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs") pod "network-metrics-daemon-9dmnl" (UID: "6acafeb4-4bd5-4969-8722-6003504206b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:14:17.728211 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:17.728012 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:14:17.728211 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:17.728038 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zdnkk for pod openshift-network-diagnostics/network-check-target-xnrdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:14:17.728211 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:17.728014 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:14:17.728211 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:17.728086 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk podName:84eade1e-4ce4-4c40-84f7-85ea4dfae29b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:49.72807073 +0000 UTC m=+66.301167513 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zdnkk" (UniqueName: "kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk") pod "network-check-target-xnrdp" (UID: "84eade1e-4ce4-4c40-84f7-85ea4dfae29b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:14:17.728211 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:17.728130 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret podName:3c21f04e-dbb0-4743-831f-462ea9ffcd9c nodeName:}" failed. No retries permitted until 2026-07-24 00:14:49.72811467 +0000 UTC m=+66.301211486 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret") pod "global-pull-secret-syncer-vncds" (UID: "3c21f04e-dbb0-4743-831f-462ea9ffcd9c") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:14:17.995958 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:17.995863 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:17.996121 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:17.996010 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vncds" podUID="3c21f04e-dbb0-4743-831f-462ea9ffcd9c" Jul 24 00:14:18.995465 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:18.995427 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:18.995465 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:18.995459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:18.995992 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:18.995577 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dmnl" podUID="6acafeb4-4bd5-4969-8722-6003504206b0" Jul 24 00:14:18.995992 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:18.995688 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrdp" podUID="84eade1e-4ce4-4c40-84f7-85ea4dfae29b" Jul 24 00:14:19.268209 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.268189 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-162.ec2.internal" event="NodeReady" Jul 24 00:14:19.268360 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.268338 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 00:14:19.310341 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.310314 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nw9wd"] Jul 24 00:14:19.311868 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.311852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:19.313479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.313455 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4r25j"] Jul 24 00:14:19.314854 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.314829 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 00:14:19.314980 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.314869 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dvjcq\"" Jul 24 00:14:19.315043 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.314979 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4r25j" Jul 24 00:14:19.315241 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.315222 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 00:14:19.317430 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.317409 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 00:14:19.317685 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.317666 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 00:14:19.317864 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.317840 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dwpqv\"" Jul 24 00:14:19.318037 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.318022 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 00:14:19.323282 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.323258 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nw9wd"] Jul 24 00:14:19.326661 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.326637 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4r25j"] Jul 24 00:14:19.427676 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.427590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:19.440542 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.440509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnv6b\" (UniqueName: \"kubernetes.io/projected/0a0428dc-cc75-43a6-8039-356a160b1b86-kube-api-access-cnv6b\") pod \"ingress-canary-4r25j\" (UID: \"0a0428dc-cc75-43a6-8039-356a160b1b86\") " pod="openshift-ingress-canary/ingress-canary-4r25j" Jul 24 00:14:19.440627 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.440582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngbv5\" (UniqueName: \"kubernetes.io/projected/03241e12-da01-4d42-abc0-30f73c4bb33b-kube-api-access-ngbv5\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:19.440627 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.440609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03241e12-da01-4d42-abc0-30f73c4bb33b-tmp-dir\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:19.440695 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.440630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03241e12-da01-4d42-abc0-30f73c4bb33b-config-volume\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:19.440695 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.440658 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:19.440753 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.440704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a0428dc-cc75-43a6-8039-356a160b1b86-cert\") pod \"ingress-canary-4r25j\" (UID: \"0a0428dc-cc75-43a6-8039-356a160b1b86\") " pod="openshift-ingress-canary/ingress-canary-4r25j" Jul 24 00:14:19.484830 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.484795 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:19.541223 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.541184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a0428dc-cc75-43a6-8039-356a160b1b86-cert\") pod \"ingress-canary-4r25j\" (UID: \"0a0428dc-cc75-43a6-8039-356a160b1b86\") " pod="openshift-ingress-canary/ingress-canary-4r25j" Jul 24 00:14:19.541412 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.541247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnv6b\" (UniqueName: \"kubernetes.io/projected/0a0428dc-cc75-43a6-8039-356a160b1b86-kube-api-access-cnv6b\") pod \"ingress-canary-4r25j\" (UID: \"0a0428dc-cc75-43a6-8039-356a160b1b86\") " pod="openshift-ingress-canary/ingress-canary-4r25j" Jul 24 00:14:19.541412 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.541289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngbv5\" (UniqueName: \"kubernetes.io/projected/03241e12-da01-4d42-abc0-30f73c4bb33b-kube-api-access-ngbv5\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:19.541412 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.541309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03241e12-da01-4d42-abc0-30f73c4bb33b-tmp-dir\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:19.541412 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.541336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03241e12-da01-4d42-abc0-30f73c4bb33b-config-volume\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:19.541412 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.541365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:19.541649 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:19.541445 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 00:14:19.541649 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:19.541502 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls podName:03241e12-da01-4d42-abc0-30f73c4bb33b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:20.041485006 +0000 UTC m=+36.614581806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls") pod "dns-default-nw9wd" (UID: "03241e12-da01-4d42-abc0-30f73c4bb33b") : secret "dns-default-metrics-tls" not found Jul 24 00:14:19.541786 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.541667 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03241e12-da01-4d42-abc0-30f73c4bb33b-tmp-dir\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:19.541910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.541889 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03241e12-da01-4d42-abc0-30f73c4bb33b-config-volume\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:19.545654 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.545630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a0428dc-cc75-43a6-8039-356a160b1b86-cert\") pod \"ingress-canary-4r25j\" (UID: \"0a0428dc-cc75-43a6-8039-356a160b1b86\") " pod="openshift-ingress-canary/ingress-canary-4r25j" Jul 24 00:14:19.551614 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.551587 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngbv5\" (UniqueName: \"kubernetes.io/projected/03241e12-da01-4d42-abc0-30f73c4bb33b-kube-api-access-ngbv5\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:19.551700 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.551596 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnv6b\" (UniqueName: \"kubernetes.io/projected/0a0428dc-cc75-43a6-8039-356a160b1b86-kube-api-access-cnv6b\") pod \"ingress-canary-4r25j\" (UID: \"0a0428dc-cc75-43a6-8039-356a160b1b86\") " pod="openshift-ingress-canary/ingress-canary-4r25j" Jul 24 00:14:19.628354 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.628317 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4r25j" Jul 24 00:14:19.753421 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.753393 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4r25j"] Jul 24 00:14:19.756992 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:19.756966 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0428dc_cc75_43a6_8039_356a160b1b86.slice/crio-089485853c29f2f7db46876c675b461c73f1e9e945f436993de14b8865c68c50 WatchSource:0}: Error finding container 089485853c29f2f7db46876c675b461c73f1e9e945f436993de14b8865c68c50: Status 404 returned error can't find the container with id 089485853c29f2f7db46876c675b461c73f1e9e945f436993de14b8865c68c50 Jul 24 00:14:19.996050 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.995964 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:19.998544 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:19.998523 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 00:14:20.045799 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:20.045759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:20.045981 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:20.045911 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 00:14:20.046023 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:20.045992 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls podName:03241e12-da01-4d42-abc0-30f73c4bb33b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:21.045975628 +0000 UTC m=+37.619072408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls") pod "dns-default-nw9wd" (UID: "03241e12-da01-4d42-abc0-30f73c4bb33b") : secret "dns-default-metrics-tls" not found Jul 24 00:14:20.164025 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:20.163978 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4r25j" event={"ID":"0a0428dc-cc75-43a6-8039-356a160b1b86","Type":"ContainerStarted","Data":"089485853c29f2f7db46876c675b461c73f1e9e945f436993de14b8865c68c50"} Jul 24 00:14:20.167543 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:20.167418 2576 generic.go:356] "Generic (PLEG): container finished" podID="d92505bf-3059-4df8-8286-9cbd3bf693c1" containerID="8a86c223e1bab8b47ca80f980f4e96ec401317595290bb664565a93289d00a40" exitCode=0 Jul 24 00:14:20.167543 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:20.167482 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vt9rk" event={"ID":"d92505bf-3059-4df8-8286-9cbd3bf693c1","Type":"ContainerDied","Data":"8a86c223e1bab8b47ca80f980f4e96ec401317595290bb664565a93289d00a40"} Jul 24 00:14:20.177399 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:20.177373 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:20.995617 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:20.995582 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:20.995793 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:20.995771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:20.999470 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:20.999169 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 00:14:20.999470 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:20.999202 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gpdt7\"" Jul 24 00:14:20.999470 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:20.999221 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 00:14:20.999470 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:20.999231 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jq7gh\"" Jul 24 00:14:20.999470 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:20.999202 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 00:14:21.052851 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:21.052824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:21.052976 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:21.052956 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 00:14:21.053032 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:21.053004 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls podName:03241e12-da01-4d42-abc0-30f73c4bb33b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:23.052991412 +0000 UTC m=+39.626088195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls") pod "dns-default-nw9wd" (UID: "03241e12-da01-4d42-abc0-30f73c4bb33b") : secret "dns-default-metrics-tls" not found Jul 24 00:14:21.173319 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:21.173284 2576 generic.go:356] "Generic (PLEG): container finished" podID="d92505bf-3059-4df8-8286-9cbd3bf693c1" containerID="26b165762cbf7a836e4f6c96dbf4590ddfc6f09efa0f70566660ab15fcde952e" exitCode=0 Jul 24 00:14:21.173481 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:21.173331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vt9rk" event={"ID":"d92505bf-3059-4df8-8286-9cbd3bf693c1","Type":"ContainerDied","Data":"26b165762cbf7a836e4f6c96dbf4590ddfc6f09efa0f70566660ab15fcde952e"} Jul 24 00:14:21.304130 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:21.304036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:21.402456 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:21.402420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:21.854448 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:21.854414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:21.954869 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:21.954835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:22.178098 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:22.178070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vt9rk" event={"ID":"d92505bf-3059-4df8-8286-9cbd3bf693c1","Type":"ContainerStarted","Data":"3dabf6a2171b029c2dd32f7b81f79348dbf8981b5785cd54cc13b6330d4010c1"} Jul 24 00:14:22.200281 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:22.200117 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vt9rk" podStartSLOduration=8.586092807 podStartE2EDuration="38.200100357s" totalImagesPullingTime="29.61400755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:22.198524984 +0000 UTC m=+38.771621786" watchObservedRunningTime="2026-07-24 00:14:22.200100357 +0000 UTC m=+38.773197161" Jul 24 00:14:22.319277 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:22.319239 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:22.419196 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:22.419061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:22.521645 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:22.521609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:22.648417 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:22.648378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" limit=1024 Jul 24 00:14:23.071404 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.071367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:23.071560 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:23.071500 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 00:14:23.071628 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:23.071559 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls podName:03241e12-da01-4d42-abc0-30f73c4bb33b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:27.071543103 +0000 UTC m=+43.644639885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls") pod "dns-default-nw9wd" (UID: "03241e12-da01-4d42-abc0-30f73c4bb33b") : secret "dns-default-metrics-tls" not found Jul 24 00:14:23.181255 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.181220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4r25j" event={"ID":"0a0428dc-cc75-43a6-8039-356a160b1b86","Type":"ContainerStarted","Data":"8ed036c6380531012c56d9a1ea1f33b668cf692ac99aad3db26e0ae35aeabe28"} Jul 24 00:14:23.196983 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.196925 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4r25j" podStartSLOduration=1.821501585 podStartE2EDuration="4.196912812s" totalImagesPullingTime="2.375411227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:23.195492065 +0000 UTC m=+39.768588867" watchObservedRunningTime="2026-07-24 00:14:23.196912812 +0000 UTC m=+39.770009613" Jul 24 00:14:23.467243 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.467169 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vrzvd"] Jul 24 00:14:23.491568 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.491543 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-r6gpd"] Jul 24 00:14:23.491710 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.491666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vrzvd" Jul 24 00:14:23.494143 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.494118 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:14:23.494255 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.494166 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-szx22\"" Jul 24 00:14:23.494898 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.494885 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 00:14:23.515633 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.515616 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vrzvd"] Jul 24 00:14:23.515713 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.515642 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-r6gpd"] Jul 24 00:14:23.515713 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.515708 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.519402 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.519379 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 00:14:23.519559 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.519437 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 00:14:23.519621 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.519562 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-nbvjd\"" Jul 24 00:14:23.519675 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.519644 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:14:23.519724 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.519671 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 00:14:23.523313 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.523286 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 00:14:23.572600 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.572568 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-564867c99-6dnpx"] Jul 24 00:14:23.595697 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.595674 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-pnv5b"] Jul 24 00:14:23.595837 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.595792 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.598330 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.598305 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 00:14:23.598330 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.598329 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 24 00:14:23.598511 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.598385 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 24 00:14:23.598511 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.598469 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-m69m6\"" Jul 24 00:14:23.598641 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.598625 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 24 00:14:23.598747 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.598730 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 24 00:14:23.598800 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.598730 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 00:14:23.619714 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.619693 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-pnv5b"] Jul 24 00:14:23.619819 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.619724 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-564867c99-6dnpx"] Jul 24 00:14:23.619888 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.619820 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.621810 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.621796 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 00:14:23.621919 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.621874 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 00:14:23.621996 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.621922 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vgt2s\"" Jul 24 00:14:23.622174 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.622156 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 00:14:23.622174 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.622173 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 00:14:23.627218 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.627202 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 00:14:23.672700 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.672672 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-stcch"] Jul 24 00:14:23.675999 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.675977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hgp2\" (UniqueName: \"kubernetes.io/projected/91bc9290-4a98-4019-8a78-d35a28b84949-kube-api-access-9hgp2\") pod \"volume-data-source-validator-549c6d54cf-vrzvd\" (UID: \"91bc9290-4a98-4019-8a78-d35a28b84949\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vrzvd" Jul 24 00:14:23.676110 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.676006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6015e731-af19-441b-8bd1-19f859ed77bb-config\") pod \"console-operator-584dcb6b54-r6gpd\" (UID: \"6015e731-af19-441b-8bd1-19f859ed77bb\") " pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.676110 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.676036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c97gk\" (UniqueName: \"kubernetes.io/projected/6015e731-af19-441b-8bd1-19f859ed77bb-kube-api-access-c97gk\") pod \"console-operator-584dcb6b54-r6gpd\" (UID: \"6015e731-af19-441b-8bd1-19f859ed77bb\") " pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.676110 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.676057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6015e731-af19-441b-8bd1-19f859ed77bb-serving-cert\") pod \"console-operator-584dcb6b54-r6gpd\" (UID: \"6015e731-af19-441b-8bd1-19f859ed77bb\") " pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.676110 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.676074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6015e731-af19-441b-8bd1-19f859ed77bb-trusted-ca\") pod \"console-operator-584dcb6b54-r6gpd\" (UID: \"6015e731-af19-441b-8bd1-19f859ed77bb\") " pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.693808 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.693788 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-v47p6"] Jul 24 00:14:23.693939 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.693923 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:23.696316 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.696296 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 00:14:23.696423 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.696335 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rwv72\"" Jul 24 00:14:23.696555 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.696541 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 00:14:23.715590 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.715564 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c4d5c85d5-52lrm"] Jul 24 00:14:23.715694 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.715677 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-v47p6" Jul 24 00:14:23.718096 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.718053 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mjlr9\"" Jul 24 00:14:23.739908 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.739883 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-stcch"] Jul 24 00:14:23.740015 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.739918 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-v47p6"] Jul 24 00:14:23.740015 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.739939 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c4d5c85d5-52lrm"] Jul 24 00:14:23.740096 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.740035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.742196 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.742179 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 00:14:23.742426 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.742414 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qlxsm\"" Jul 24 00:14:23.742565 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.742548 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 00:14:23.742613 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.742576 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 00:14:23.747746 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.747729 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 00:14:23.776544 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6015e731-af19-441b-8bd1-19f859ed77bb-trusted-ca\") pod \"console-operator-584dcb6b54-r6gpd\" (UID: \"6015e731-af19-441b-8bd1-19f859ed77bb\") " pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.776623 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-stats-auth\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.776623 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-snapshots\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.776689 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4655l\" (UniqueName: \"kubernetes.io/projected/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-kube-api-access-4655l\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.776689 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-tmp\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.776750 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-serving-cert\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.776784 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.776819 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hgp2\" (UniqueName: \"kubernetes.io/projected/91bc9290-4a98-4019-8a78-d35a28b84949-kube-api-access-9hgp2\") pod \"volume-data-source-validator-549c6d54cf-vrzvd\" (UID: \"91bc9290-4a98-4019-8a78-d35a28b84949\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vrzvd" Jul 24 00:14:23.776819 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6015e731-af19-441b-8bd1-19f859ed77bb-config\") pod \"console-operator-584dcb6b54-r6gpd\" (UID: \"6015e731-af19-441b-8bd1-19f859ed77bb\") " pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.776885 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-default-certificate\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.776885 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qsjl\" (UniqueName: \"kubernetes.io/projected/7acb8462-421f-47e3-8bed-596f4f2d4d15-kube-api-access-9qsjl\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.776885 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.777036 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-service-ca-bundle\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.777036 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c97gk\" (UniqueName: \"kubernetes.io/projected/6015e731-af19-441b-8bd1-19f859ed77bb-kube-api-access-c97gk\") pod \"console-operator-584dcb6b54-r6gpd\" (UID: \"6015e731-af19-441b-8bd1-19f859ed77bb\") " pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.777036 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.776976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.777036 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.777000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6015e731-af19-441b-8bd1-19f859ed77bb-serving-cert\") pod \"console-operator-584dcb6b54-r6gpd\" (UID: \"6015e731-af19-441b-8bd1-19f859ed77bb\") " pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.777383 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.777364 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6015e731-af19-441b-8bd1-19f859ed77bb-trusted-ca\") pod \"console-operator-584dcb6b54-r6gpd\" (UID: \"6015e731-af19-441b-8bd1-19f859ed77bb\") " pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.777512 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.777387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6015e731-af19-441b-8bd1-19f859ed77bb-config\") pod \"console-operator-584dcb6b54-r6gpd\" (UID: \"6015e731-af19-441b-8bd1-19f859ed77bb\") " pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.780646 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.780629 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6015e731-af19-441b-8bd1-19f859ed77bb-serving-cert\") pod \"console-operator-584dcb6b54-r6gpd\" (UID: \"6015e731-af19-441b-8bd1-19f859ed77bb\") " pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.784978 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.784943 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hgp2\" (UniqueName: \"kubernetes.io/projected/91bc9290-4a98-4019-8a78-d35a28b84949-kube-api-access-9hgp2\") pod \"volume-data-source-validator-549c6d54cf-vrzvd\" (UID: \"91bc9290-4a98-4019-8a78-d35a28b84949\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vrzvd" Jul 24 00:14:23.785280 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.785263 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c97gk\" (UniqueName: \"kubernetes.io/projected/6015e731-af19-441b-8bd1-19f859ed77bb-kube-api-access-c97gk\") pod \"console-operator-584dcb6b54-r6gpd\" (UID: \"6015e731-af19-441b-8bd1-19f859ed77bb\") " pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.801000 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.800979 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vrzvd" Jul 24 00:14:23.826342 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.826325 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:23.878401 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.878368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.878528 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.878417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qsjl\" (UniqueName: \"kubernetes.io/projected/7acb8462-421f-47e3-8bed-596f4f2d4d15-kube-api-access-9qsjl\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.878528 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.878445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.878625 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:23.878583 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 00:14:23.878716 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:23.878704 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs podName:7acb8462-421f-47e3-8bed-596f4f2d4d15 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:24.378662369 +0000 UTC m=+40.951759169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs") pod "router-default-564867c99-6dnpx" (UID: "7acb8462-421f-47e3-8bed-596f4f2d4d15") : secret "router-metrics-certs-default" not found Jul 24 00:14:23.879032 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-snapshots\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.879093 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-image-registry-private-configuration\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.879133 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-default-certificate\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.879206 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5tsx\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-kube-api-access-w5tsx\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.879467 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879445 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dabcdd63-5694-4208-826b-6682b43c826d-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-stcch\" (UID: \"dabcdd63-5694-4208-826b-6682b43c826d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:23.879545 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-ca-trust-extracted\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.879545 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-trusted-ca\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.879646 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-tmp\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.879646 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-serving-cert\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.879646 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879584 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.879646 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.879846 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-stcch\" (UID: \"dabcdd63-5694-4208-826b-6682b43c826d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:23.879846 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-service-ca-bundle\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.879846 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.879846 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879815 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-snapshots\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.879846 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-certificates\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.880125 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879895 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-installation-pull-secrets\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.880125 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879893 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-tmp\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.880125 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-stats-auth\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.880125 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.879999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjf99\" (UniqueName: \"kubernetes.io/projected/d6a61aef-01a2-47ba-9503-fd05836ac40c-kube-api-access-mjf99\") pod \"network-check-source-7d88cb55d8-v47p6\" (UID: \"d6a61aef-01a2-47ba-9503-fd05836ac40c\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-v47p6" Jul 24 00:14:23.880125 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:23.880113 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle podName:7acb8462-421f-47e3-8bed-596f4f2d4d15 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:24.380092354 +0000 UTC m=+40.953189158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle") pod "router-default-564867c99-6dnpx" (UID: "7acb8462-421f-47e3-8bed-596f4f2d4d15") : configmap references non-existent config key: service-ca.crt Jul 24 00:14:23.880379 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.880142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-bound-sa-token\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.880379 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.880176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4655l\" (UniqueName: \"kubernetes.io/projected/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-kube-api-access-4655l\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.880379 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.880329 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-service-ca-bundle\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.882159 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.882141 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-serving-cert\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.882501 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.882485 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-default-certificate\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.882847 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.882828 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-stats-auth\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.888472 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.887724 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qsjl\" (UniqueName: \"kubernetes.io/projected/7acb8462-421f-47e3-8bed-596f4f2d4d15-kube-api-access-9qsjl\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:23.888472 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.887771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4655l\" (UniqueName: \"kubernetes.io/projected/86ed4490-c0cd-45a0-bfe0-ba5730e4b621-kube-api-access-4655l\") pod \"insights-operator-5885fdb759-pnv5b\" (UID: \"86ed4490-c0cd-45a0-bfe0-ba5730e4b621\") " pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.928598 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.928330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-pnv5b" Jul 24 00:14:23.973204 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.973103 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vrzvd"] Jul 24 00:14:23.976267 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:23.976234 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91bc9290_4a98_4019_8a78_d35a28b84949.slice/crio-2c6e82111370f70b504eb229e6265c63f59e40cd04e2c41106ccb42dee96d8bd WatchSource:0}: Error finding container 2c6e82111370f70b504eb229e6265c63f59e40cd04e2c41106ccb42dee96d8bd: Status 404 returned error can't find the container with id 2c6e82111370f70b504eb229e6265c63f59e40cd04e2c41106ccb42dee96d8bd Jul 24 00:14:23.981224 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.981201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5tsx\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-kube-api-access-w5tsx\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.981314 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.981233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dabcdd63-5694-4208-826b-6682b43c826d-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-stcch\" (UID: \"dabcdd63-5694-4208-826b-6682b43c826d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:23.981314 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.981254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-ca-trust-extracted\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.981314 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.981270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-trusted-ca\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.981473 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.981313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-stcch\" (UID: \"dabcdd63-5694-4208-826b-6682b43c826d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:23.981473 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.981359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.981473 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.981384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-certificates\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.981473 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.981414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-installation-pull-secrets\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.981473 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.981448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjf99\" (UniqueName: \"kubernetes.io/projected/d6a61aef-01a2-47ba-9503-fd05836ac40c-kube-api-access-mjf99\") pod \"network-check-source-7d88cb55d8-v47p6\" (UID: \"d6a61aef-01a2-47ba-9503-fd05836ac40c\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-v47p6" Jul 24 00:14:23.981695 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.981473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-bound-sa-token\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.981695 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:23.981540 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 00:14:23.981695 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.981560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-image-registry-private-configuration\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.981695 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:23.981561 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 00:14:23.981695 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:23.981617 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert podName:dabcdd63-5694-4208-826b-6682b43c826d nodeName:}" failed. No retries permitted until 2026-07-24 00:14:24.481594333 +0000 UTC m=+41.054691128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-stcch" (UID: "dabcdd63-5694-4208-826b-6682b43c826d") : secret "networking-console-plugin-cert" not found Jul 24 00:14:23.981695 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:23.981623 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c4d5c85d5-52lrm: secret "image-registry-tls" not found Jul 24 00:14:23.981695 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:23.981669 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls podName:9ae9d4a5-4472-4cc3-b83c-626d061d2b55 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:24.481654659 +0000 UTC m=+41.054751459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls") pod "image-registry-5c4d5c85d5-52lrm" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55") : secret "image-registry-tls" not found Jul 24 00:14:23.981695 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.981664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-ca-trust-extracted\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.982375 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.982054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dabcdd63-5694-4208-826b-6682b43c826d-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-stcch\" (UID: \"dabcdd63-5694-4208-826b-6682b43c826d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:23.982375 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.982154 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-certificates\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.982535 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.982478 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-trusted-ca\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.984821 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.984780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-installation-pull-secrets\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.984821 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.984792 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-image-registry-private-configuration\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.989176 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.989153 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-r6gpd"] Jul 24 00:14:23.992101 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:23.992079 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6015e731_af19_441b_8bd1_19f859ed77bb.slice/crio-d8b73276589faa1d893212db85cc40ddc874319f53f877bcf4fdba3eda2d9d11 WatchSource:0}: Error finding container d8b73276589faa1d893212db85cc40ddc874319f53f877bcf4fdba3eda2d9d11: Status 404 returned error can't find the container with id d8b73276589faa1d893212db85cc40ddc874319f53f877bcf4fdba3eda2d9d11 Jul 24 00:14:23.997356 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.997266 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjf99\" (UniqueName: \"kubernetes.io/projected/d6a61aef-01a2-47ba-9503-fd05836ac40c-kube-api-access-mjf99\") pod \"network-check-source-7d88cb55d8-v47p6\" (UID: \"d6a61aef-01a2-47ba-9503-fd05836ac40c\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-v47p6" Jul 24 00:14:23.997442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.997391 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5tsx\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-kube-api-access-w5tsx\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:23.998570 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:23.998546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-bound-sa-token\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:24.022793 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:24.022763 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-v47p6" Jul 24 00:14:24.088143 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:24.088115 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-pnv5b"] Jul 24 00:14:24.092370 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:24.092318 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ed4490_c0cd_45a0_bfe0_ba5730e4b621.slice/crio-08ff3d4e674b3938768f43ecdbfc5c2b4ca0399f0cf7a3eb308cb14286ec4424 WatchSource:0}: Error finding container 08ff3d4e674b3938768f43ecdbfc5c2b4ca0399f0cf7a3eb308cb14286ec4424: Status 404 returned error can't find the container with id 08ff3d4e674b3938768f43ecdbfc5c2b4ca0399f0cf7a3eb308cb14286ec4424 Jul 24 00:14:24.156592 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:24.156561 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-v47p6"] Jul 24 00:14:24.160020 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:24.159992 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a61aef_01a2_47ba_9503_fd05836ac40c.slice/crio-b676e3459c138d07f7378c956bead6135110fea6c393015369e0aca7dcd79c46 WatchSource:0}: Error finding container b676e3459c138d07f7378c956bead6135110fea6c393015369e0aca7dcd79c46: Status 404 returned error can't find the container with id b676e3459c138d07f7378c956bead6135110fea6c393015369e0aca7dcd79c46 Jul 24 00:14:24.184808 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:24.184772 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vrzvd" event={"ID":"91bc9290-4a98-4019-8a78-d35a28b84949","Type":"ContainerStarted","Data":"2c6e82111370f70b504eb229e6265c63f59e40cd04e2c41106ccb42dee96d8bd"} Jul 24 00:14:24.185796 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:24.185771 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-v47p6" event={"ID":"d6a61aef-01a2-47ba-9503-fd05836ac40c","Type":"ContainerStarted","Data":"b676e3459c138d07f7378c956bead6135110fea6c393015369e0aca7dcd79c46"} Jul 24 00:14:24.186721 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:24.186692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-pnv5b" event={"ID":"86ed4490-c0cd-45a0-bfe0-ba5730e4b621","Type":"ContainerStarted","Data":"08ff3d4e674b3938768f43ecdbfc5c2b4ca0399f0cf7a3eb308cb14286ec4424"} Jul 24 00:14:24.187687 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:24.187667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" event={"ID":"6015e731-af19-441b-8bd1-19f859ed77bb","Type":"ContainerStarted","Data":"d8b73276589faa1d893212db85cc40ddc874319f53f877bcf4fdba3eda2d9d11"} Jul 24 00:14:24.384635 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:24.384592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:24.384835 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:24.384659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:24.384835 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:24.384739 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 00:14:24.384835 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:24.384801 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle podName:7acb8462-421f-47e3-8bed-596f4f2d4d15 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:25.384788326 +0000 UTC m=+41.957885107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle") pod "router-default-564867c99-6dnpx" (UID: "7acb8462-421f-47e3-8bed-596f4f2d4d15") : configmap references non-existent config key: service-ca.crt Jul 24 00:14:24.384835 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:24.384817 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs podName:7acb8462-421f-47e3-8bed-596f4f2d4d15 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:25.384811306 +0000 UTC m=+41.957908086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs") pod "router-default-564867c99-6dnpx" (UID: "7acb8462-421f-47e3-8bed-596f4f2d4d15") : secret "router-metrics-certs-default" not found Jul 24 00:14:24.485224 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:24.485189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-stcch\" (UID: \"dabcdd63-5694-4208-826b-6682b43c826d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:24.485390 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:24.485243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:24.485390 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:24.485335 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 00:14:24.485472 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:24.485400 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert podName:dabcdd63-5694-4208-826b-6682b43c826d nodeName:}" failed. No retries permitted until 2026-07-24 00:14:25.485383381 +0000 UTC m=+42.058480162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-stcch" (UID: "dabcdd63-5694-4208-826b-6682b43c826d") : secret "networking-console-plugin-cert" not found Jul 24 00:14:24.485472 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:24.485400 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 00:14:24.485472 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:24.485418 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c4d5c85d5-52lrm: secret "image-registry-tls" not found Jul 24 00:14:24.485472 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:24.485458 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls podName:9ae9d4a5-4472-4cc3-b83c-626d061d2b55 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:25.485445188 +0000 UTC m=+42.058541972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls") pod "image-registry-5c4d5c85d5-52lrm" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55") : secret "image-registry-tls" not found Jul 24 00:14:25.394694 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:25.394653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:25.395271 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:25.394745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:25.395271 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:25.394994 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle podName:7acb8462-421f-47e3-8bed-596f4f2d4d15 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:27.394971564 +0000 UTC m=+43.968068346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle") pod "router-default-564867c99-6dnpx" (UID: "7acb8462-421f-47e3-8bed-596f4f2d4d15") : configmap references non-existent config key: service-ca.crt Jul 24 00:14:25.395271 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:25.395064 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 00:14:25.395271 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:25.395137 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs podName:7acb8462-421f-47e3-8bed-596f4f2d4d15 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:27.39511726 +0000 UTC m=+43.968214050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs") pod "router-default-564867c99-6dnpx" (UID: "7acb8462-421f-47e3-8bed-596f4f2d4d15") : secret "router-metrics-certs-default" not found Jul 24 00:14:25.496291 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:25.496211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:25.496494 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:25.496374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-stcch\" (UID: \"dabcdd63-5694-4208-826b-6682b43c826d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:25.496557 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:25.496494 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 00:14:25.496611 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:25.496556 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert podName:dabcdd63-5694-4208-826b-6682b43c826d nodeName:}" failed. No retries permitted until 2026-07-24 00:14:27.496537817 +0000 UTC m=+44.069634599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-stcch" (UID: "dabcdd63-5694-4208-826b-6682b43c826d") : secret "networking-console-plugin-cert" not found Jul 24 00:14:25.496837 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:25.496809 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 00:14:25.496837 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:25.496832 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c4d5c85d5-52lrm: secret "image-registry-tls" not found Jul 24 00:14:25.497052 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:25.496882 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls podName:9ae9d4a5-4472-4cc3-b83c-626d061d2b55 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:27.496866944 +0000 UTC m=+44.069963743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls") pod "image-registry-5c4d5c85d5-52lrm" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55") : secret "image-registry-tls" not found Jul 24 00:14:26.194858 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:26.194821 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vrzvd" event={"ID":"91bc9290-4a98-4019-8a78-d35a28b84949","Type":"ContainerStarted","Data":"35070d68f994c94989a52f3f96a37d4f98652c785a293241d67bad5455bc6a88"} Jul 24 00:14:27.110390 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:27.110356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:27.110727 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:27.110508 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 00:14:27.110727 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:27.110572 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls podName:03241e12-da01-4d42-abc0-30f73c4bb33b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:35.110556893 +0000 UTC m=+51.683653677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls") pod "dns-default-nw9wd" (UID: "03241e12-da01-4d42-abc0-30f73c4bb33b") : secret "dns-default-metrics-tls" not found Jul 24 00:14:27.412748 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:27.412668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:27.412883 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:27.412743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:27.412883 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:27.412828 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 00:14:27.412969 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:27.412897 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs podName:7acb8462-421f-47e3-8bed-596f4f2d4d15 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:31.412879319 +0000 UTC m=+47.985976113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs") pod "router-default-564867c99-6dnpx" (UID: "7acb8462-421f-47e3-8bed-596f4f2d4d15") : secret "router-metrics-certs-default" not found Jul 24 00:14:27.412969 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:27.412913 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle podName:7acb8462-421f-47e3-8bed-596f4f2d4d15 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:31.412905644 +0000 UTC m=+47.986002424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle") pod "router-default-564867c99-6dnpx" (UID: "7acb8462-421f-47e3-8bed-596f4f2d4d15") : configmap references non-existent config key: service-ca.crt Jul 24 00:14:27.513579 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:27.513524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-stcch\" (UID: \"dabcdd63-5694-4208-826b-6682b43c826d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:27.513769 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:27.513599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:27.513769 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:27.513694 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 00:14:27.513769 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:27.513743 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 00:14:27.513769 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:27.513762 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c4d5c85d5-52lrm: secret "image-registry-tls" not found Jul 24 00:14:27.513769 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:27.513771 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert podName:dabcdd63-5694-4208-826b-6682b43c826d nodeName:}" failed. No retries permitted until 2026-07-24 00:14:31.513753836 +0000 UTC m=+48.086850617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-stcch" (UID: "dabcdd63-5694-4208-826b-6682b43c826d") : secret "networking-console-plugin-cert" not found Jul 24 00:14:27.514039 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:27.513813 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls podName:9ae9d4a5-4472-4cc3-b83c-626d061d2b55 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:31.513793766 +0000 UTC m=+48.086890548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls") pod "image-registry-5c4d5c85d5-52lrm" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55") : secret "image-registry-tls" not found Jul 24 00:14:29.204288 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.204255 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/0.log" Jul 24 00:14:29.204805 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.204300 2576 generic.go:356] "Generic (PLEG): container finished" podID="6015e731-af19-441b-8bd1-19f859ed77bb" containerID="edb99a27961af1be439599ea281845118bfed652278bedf0a354577a44330d97" exitCode=255 Jul 24 00:14:29.204805 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.204412 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" event={"ID":"6015e731-af19-441b-8bd1-19f859ed77bb","Type":"ContainerDied","Data":"edb99a27961af1be439599ea281845118bfed652278bedf0a354577a44330d97"} Jul 24 00:14:29.204805 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.204745 2576 scope.go:117] "RemoveContainer" containerID="edb99a27961af1be439599ea281845118bfed652278bedf0a354577a44330d97" Jul 24 00:14:29.206373 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.206342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-v47p6" event={"ID":"d6a61aef-01a2-47ba-9503-fd05836ac40c","Type":"ContainerStarted","Data":"57f85646643f3f9c3ed369ab3c05592128bb11040afd2651f2dbb374178eb8a4"} Jul 24 00:14:29.207966 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.207929 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-pnv5b" event={"ID":"86ed4490-c0cd-45a0-bfe0-ba5730e4b621","Type":"ContainerStarted","Data":"60da693b3be2fa1bf47cce9b1513cdda251ed47476797d97a4c4c7a473d82fd1"} Jul 24 00:14:29.219478 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.219425 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vrzvd" podStartSLOduration=4.527675815 podStartE2EDuration="6.219406054s" totalImagesPullingTime="1.691730239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:26.209449092 +0000 UTC m=+42.782545896" watchObservedRunningTime="2026-07-24 00:14:29.219406054 +0000 UTC m=+45.792502862" Jul 24 00:14:29.234062 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.234009 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-pnv5b" podStartSLOduration=1.611944556 podStartE2EDuration="6.233986719s" totalImagesPullingTime="4.622042163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:29.232440009 +0000 UTC m=+45.805536814" watchObservedRunningTime="2026-07-24 00:14:29.233986719 +0000 UTC m=+45.807083522" Jul 24 00:14:29.247302 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.247245 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-v47p6" podStartSLOduration=1.6852401000000001 podStartE2EDuration="6.247222706s" totalImagesPullingTime="4.561982606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:29.245920113 +0000 UTC m=+45.819016912" watchObservedRunningTime="2026-07-24 00:14:29.247222706 +0000 UTC m=+45.820319507" Jul 24 00:14:29.586939 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.586908 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k66cd"] Jul 24 00:14:29.612532 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.612496 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k66cd"] Jul 24 00:14:29.612672 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.612616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.614995 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.614975 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 00:14:29.615136 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.615006 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mtjsg\"" Jul 24 00:14:29.615324 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.615308 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 00:14:29.634132 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.634113 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.634221 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.634169 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04e9e66a-b4ae-4fe2-a876-440487e20e2b-crio-socket\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.634221 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.634188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e9e66a-b4ae-4fe2-a876-440487e20e2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.634288 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.634267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnh7k\" (UniqueName: \"kubernetes.io/projected/04e9e66a-b4ae-4fe2-a876-440487e20e2b-kube-api-access-pnh7k\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.634325 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.634296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04e9e66a-b4ae-4fe2-a876-440487e20e2b-data-volume\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.734774 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.734743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnh7k\" (UniqueName: \"kubernetes.io/projected/04e9e66a-b4ae-4fe2-a876-440487e20e2b-kube-api-access-pnh7k\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.734890 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.734777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04e9e66a-b4ae-4fe2-a876-440487e20e2b-data-volume\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.734986 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.734962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.735079 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.735063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04e9e66a-b4ae-4fe2-a876-440487e20e2b-crio-socket\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.735123 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:29.735089 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 00:14:29.735189 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.735126 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04e9e66a-b4ae-4fe2-a876-440487e20e2b-data-volume\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.735189 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.735096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e9e66a-b4ae-4fe2-a876-440487e20e2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.735189 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:29.735181 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 00:14:29.735316 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:29.735193 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls podName:04e9e66a-b4ae-4fe2-a876-440487e20e2b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:30.235171913 +0000 UTC m=+46.808268699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k66cd" (UID: "04e9e66a-b4ae-4fe2-a876-440487e20e2b") : secret "insights-runtime-extractor-tls" not found Jul 24 00:14:29.735316 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:29.735233 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e9e66a-b4ae-4fe2-a876-440487e20e2b-kube-rbac-proxy-cm podName:04e9e66a-b4ae-4fe2-a876-440487e20e2b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:30.235216391 +0000 UTC m=+46.808313188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04e9e66a-b4ae-4fe2-a876-440487e20e2b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k66cd" (UID: "04e9e66a-b4ae-4fe2-a876-440487e20e2b") : configmap "kube-rbac-proxy" not found Jul 24 00:14:29.735405 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.735312 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04e9e66a-b4ae-4fe2-a876-440487e20e2b-crio-socket\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:29.743082 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:29.743063 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnh7k\" (UniqueName: \"kubernetes.io/projected/04e9e66a-b4ae-4fe2-a876-440487e20e2b-kube-api-access-pnh7k\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:30.213199 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:30.213175 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:14:30.213611 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:30.213597 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/0.log" Jul 24 00:14:30.213656 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:30.213633 2576 generic.go:356] "Generic (PLEG): container finished" podID="6015e731-af19-441b-8bd1-19f859ed77bb" containerID="75112ebd7a6c885337c755421025a9e5165e03468a86d6de7397865be7187a63" exitCode=255 Jul 24 00:14:30.213756 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:30.213725 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" event={"ID":"6015e731-af19-441b-8bd1-19f859ed77bb","Type":"ContainerDied","Data":"75112ebd7a6c885337c755421025a9e5165e03468a86d6de7397865be7187a63"} Jul 24 00:14:30.213820 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:30.213808 2576 scope.go:117] "RemoveContainer" containerID="edb99a27961af1be439599ea281845118bfed652278bedf0a354577a44330d97" Jul 24 00:14:30.214056 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:30.214040 2576 scope.go:117] "RemoveContainer" containerID="75112ebd7a6c885337c755421025a9e5165e03468a86d6de7397865be7187a63" Jul 24 00:14:30.214236 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:30.214217 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-r6gpd_openshift-console-operator(6015e731-af19-441b-8bd1-19f859ed77bb)\"" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" podUID="6015e731-af19-441b-8bd1-19f859ed77bb" Jul 24 00:14:30.239177 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:30.239130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:30.239488 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:30.239458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e9e66a-b4ae-4fe2-a876-440487e20e2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:30.239675 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:30.239568 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 00:14:30.239793 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:30.239598 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 00:14:30.239793 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:30.239724 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e9e66a-b4ae-4fe2-a876-440487e20e2b-kube-rbac-proxy-cm podName:04e9e66a-b4ae-4fe2-a876-440487e20e2b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:31.239701262 +0000 UTC m=+47.812798070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04e9e66a-b4ae-4fe2-a876-440487e20e2b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k66cd" (UID: "04e9e66a-b4ae-4fe2-a876-440487e20e2b") : configmap "kube-rbac-proxy" not found Jul 24 00:14:30.239793 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:30.239787 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls podName:04e9e66a-b4ae-4fe2-a876-440487e20e2b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:31.239770046 +0000 UTC m=+47.812866832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k66cd" (UID: "04e9e66a-b4ae-4fe2-a876-440487e20e2b") : secret "insights-runtime-extractor-tls" not found Jul 24 00:14:31.217589 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:31.217563 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:14:31.217991 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:31.217896 2576 scope.go:117] "RemoveContainer" containerID="75112ebd7a6c885337c755421025a9e5165e03468a86d6de7397865be7187a63" Jul 24 00:14:31.218092 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:31.218073 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-r6gpd_openshift-console-operator(6015e731-af19-441b-8bd1-19f859ed77bb)\"" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" podUID="6015e731-af19-441b-8bd1-19f859ed77bb" Jul 24 00:14:31.249705 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:31.249673 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e9e66a-b4ae-4fe2-a876-440487e20e2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:31.249940 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:31.249911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:31.250089 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:31.250058 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 00:14:31.250157 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:31.250104 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls podName:04e9e66a-b4ae-4fe2-a876-440487e20e2b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:33.250090871 +0000 UTC m=+49.823187651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k66cd" (UID: "04e9e66a-b4ae-4fe2-a876-440487e20e2b") : secret "insights-runtime-extractor-tls" not found Jul 24 00:14:31.250323 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:31.250275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e9e66a-b4ae-4fe2-a876-440487e20e2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:31.452081 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:31.452038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:31.452277 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:31.452183 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 00:14:31.452277 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:31.452227 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:31.452277 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:31.452242 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs podName:7acb8462-421f-47e3-8bed-596f4f2d4d15 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:39.452227142 +0000 UTC m=+56.025323923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs") pod "router-default-564867c99-6dnpx" (UID: "7acb8462-421f-47e3-8bed-596f4f2d4d15") : secret "router-metrics-certs-default" not found Jul 24 00:14:31.452430 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:31.452327 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle podName:7acb8462-421f-47e3-8bed-596f4f2d4d15 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:39.452310282 +0000 UTC m=+56.025407087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle") pod "router-default-564867c99-6dnpx" (UID: "7acb8462-421f-47e3-8bed-596f4f2d4d15") : configmap references non-existent config key: service-ca.crt Jul 24 00:14:31.552983 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:31.552929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-stcch\" (UID: \"dabcdd63-5694-4208-826b-6682b43c826d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:31.553156 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:31.553008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:31.553156 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:31.553050 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 00:14:31.553156 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:31.553102 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 00:14:31.553156 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:31.553112 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c4d5c85d5-52lrm: secret "image-registry-tls" not found Jul 24 00:14:31.553156 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:31.553113 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert podName:dabcdd63-5694-4208-826b-6682b43c826d nodeName:}" failed. No retries permitted until 2026-07-24 00:14:39.553098275 +0000 UTC m=+56.126195067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-stcch" (UID: "dabcdd63-5694-4208-826b-6682b43c826d") : secret "networking-console-plugin-cert" not found Jul 24 00:14:31.553337 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:31.553161 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls podName:9ae9d4a5-4472-4cc3-b83c-626d061d2b55 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:39.553143531 +0000 UTC m=+56.126240311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls") pod "image-registry-5c4d5c85d5-52lrm" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55") : secret "image-registry-tls" not found Jul 24 00:14:31.776126 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:31.776099 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8lgmt_301f4b24-c1c9-4754-ae99-e606ac261c2e/dns-node-resolver/0.log" Jul 24 00:14:32.776603 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:32.776577 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h6xgq_f1261605-62c9-473d-bf74-156b31afd9c6/node-ca/0.log" Jul 24 00:14:33.268670 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:33.268630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:33.268846 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:33.268782 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 00:14:33.268895 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:33.268847 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls podName:04e9e66a-b4ae-4fe2-a876-440487e20e2b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:37.268831415 +0000 UTC m=+53.841928199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k66cd" (UID: "04e9e66a-b4ae-4fe2-a876-440487e20e2b") : secret "insights-runtime-extractor-tls" not found Jul 24 00:14:33.575006 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:33.574969 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4r25j_0a0428dc-cc75-43a6-8039-356a160b1b86/serve-healthcheck-canary/0.log" Jul 24 00:14:33.827139 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:33.827045 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:33.827139 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:33.827108 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:33.827552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:33.827433 2576 scope.go:117] "RemoveContainer" containerID="75112ebd7a6c885337c755421025a9e5165e03468a86d6de7397865be7187a63" Jul 24 00:14:33.827598 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:33.827578 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-r6gpd_openshift-console-operator(6015e731-af19-441b-8bd1-19f859ed77bb)\"" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" podUID="6015e731-af19-441b-8bd1-19f859ed77bb" Jul 24 00:14:35.186793 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:35.186758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:35.187215 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:35.186921 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 00:14:35.187215 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:35.186996 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls podName:03241e12-da01-4d42-abc0-30f73c4bb33b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:51.186975936 +0000 UTC m=+67.760072716 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls") pod "dns-default-nw9wd" (UID: "03241e12-da01-4d42-abc0-30f73c4bb33b") : secret "dns-default-metrics-tls" not found Jul 24 00:14:37.306498 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:37.306451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:37.306871 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:37.306609 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 00:14:37.306871 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:37.306687 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls podName:04e9e66a-b4ae-4fe2-a876-440487e20e2b nodeName:}" failed. No retries permitted until 2026-07-24 00:14:45.306669368 +0000 UTC m=+61.879766161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k66cd" (UID: "04e9e66a-b4ae-4fe2-a876-440487e20e2b") : secret "insights-runtime-extractor-tls" not found Jul 24 00:14:39.526204 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:39.526163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:39.526710 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:39.526224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:39.526773 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:39.526756 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7acb8462-421f-47e3-8bed-596f4f2d4d15-service-ca-bundle\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:39.528487 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:39.528464 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb8462-421f-47e3-8bed-596f4f2d4d15-metrics-certs\") pod \"router-default-564867c99-6dnpx\" (UID: \"7acb8462-421f-47e3-8bed-596f4f2d4d15\") " pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:39.627258 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:39.627192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-stcch\" (UID: \"dabcdd63-5694-4208-826b-6682b43c826d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:39.627476 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:39.627289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:39.627476 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:39.627366 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 00:14:39.627476 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:14:39.627449 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert podName:dabcdd63-5694-4208-826b-6682b43c826d nodeName:}" failed. No retries permitted until 2026-07-24 00:14:55.627428714 +0000 UTC m=+72.200525519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-stcch" (UID: "dabcdd63-5694-4208-826b-6682b43c826d") : secret "networking-console-plugin-cert" not found Jul 24 00:14:39.629758 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:39.629732 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls\") pod \"image-registry-5c4d5c85d5-52lrm\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:39.648171 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:39.648142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:39.769745 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:39.769659 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c4d5c85d5-52lrm"] Jul 24 00:14:39.772506 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:39.772477 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae9d4a5_4472_4cc3_b83c_626d061d2b55.slice/crio-58bc06fc1953cbbb36bde4ffa12d13ad42d4e272f44ad3af96fb706e1642406c WatchSource:0}: Error finding container 58bc06fc1953cbbb36bde4ffa12d13ad42d4e272f44ad3af96fb706e1642406c: Status 404 returned error can't find the container with id 58bc06fc1953cbbb36bde4ffa12d13ad42d4e272f44ad3af96fb706e1642406c Jul 24 00:14:39.804729 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:39.804702 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:39.928097 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:39.928067 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-564867c99-6dnpx"] Jul 24 00:14:39.931032 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:39.931002 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acb8462_421f_47e3_8bed_596f4f2d4d15.slice/crio-417c20a50f99f7bb4b2aad87b6534424299e37301c6e8225f9a921946c40b622 WatchSource:0}: Error finding container 417c20a50f99f7bb4b2aad87b6534424299e37301c6e8225f9a921946c40b622: Status 404 returned error can't find the container with id 417c20a50f99f7bb4b2aad87b6534424299e37301c6e8225f9a921946c40b622 Jul 24 00:14:40.242550 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:40.242516 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" event={"ID":"9ae9d4a5-4472-4cc3-b83c-626d061d2b55","Type":"ContainerStarted","Data":"0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875"} Jul 24 00:14:40.242722 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:40.242564 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" event={"ID":"9ae9d4a5-4472-4cc3-b83c-626d061d2b55","Type":"ContainerStarted","Data":"58bc06fc1953cbbb36bde4ffa12d13ad42d4e272f44ad3af96fb706e1642406c"} Jul 24 00:14:40.242722 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:40.242622 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:14:40.243934 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:40.243911 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-564867c99-6dnpx" event={"ID":"7acb8462-421f-47e3-8bed-596f4f2d4d15","Type":"ContainerStarted","Data":"cc39e682dae148318fe2882968f586c6e3c07038ee3facaf85ff75ed3192566e"} Jul 24 00:14:40.244059 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:40.243964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-564867c99-6dnpx" event={"ID":"7acb8462-421f-47e3-8bed-596f4f2d4d15","Type":"ContainerStarted","Data":"417c20a50f99f7bb4b2aad87b6534424299e37301c6e8225f9a921946c40b622"} Jul 24 00:14:40.259613 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:40.259570 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" podStartSLOduration=17.259556298 podStartE2EDuration="17.259556298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:40.258576826 +0000 UTC m=+56.831673626" watchObservedRunningTime="2026-07-24 00:14:40.259556298 +0000 UTC m=+56.832653101" Jul 24 00:14:40.276603 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:40.276562 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-564867c99-6dnpx" podStartSLOduration=17.27654938 podStartE2EDuration="17.27654938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:40.275815022 +0000 UTC m=+56.848911816" watchObservedRunningTime="2026-07-24 00:14:40.27654938 +0000 UTC m=+56.849646182" Jul 24 00:14:40.805848 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:40.805812 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:40.808160 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:40.808136 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:41.247188 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:41.247076 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:41.248284 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:41.248263 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-564867c99-6dnpx" Jul 24 00:14:43.154467 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:43.154440 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dtgbd" Jul 24 00:14:45.379736 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:45.379700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:45.381896 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:45.381878 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e9e66a-b4ae-4fe2-a876-440487e20e2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k66cd\" (UID: \"04e9e66a-b4ae-4fe2-a876-440487e20e2b\") " pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:45.523285 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:45.523257 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mtjsg\"" Jul 24 00:14:45.531212 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:45.531172 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k66cd" Jul 24 00:14:45.652455 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:45.652383 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k66cd"] Jul 24 00:14:45.655906 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:45.655874 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e9e66a_b4ae_4fe2_a876_440487e20e2b.slice/crio-52d576d7fae7d1fbe4ea3318ea8ec8a31cb23dcf616e6f27d47e07dbb3f21f61 WatchSource:0}: Error finding container 52d576d7fae7d1fbe4ea3318ea8ec8a31cb23dcf616e6f27d47e07dbb3f21f61: Status 404 returned error can't find the container with id 52d576d7fae7d1fbe4ea3318ea8ec8a31cb23dcf616e6f27d47e07dbb3f21f61 Jul 24 00:14:46.266201 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:46.266116 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k66cd" event={"ID":"04e9e66a-b4ae-4fe2-a876-440487e20e2b","Type":"ContainerStarted","Data":"52d576d7fae7d1fbe4ea3318ea8ec8a31cb23dcf616e6f27d47e07dbb3f21f61"} Jul 24 00:14:47.996077 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:47.996040 2576 scope.go:117] "RemoveContainer" containerID="75112ebd7a6c885337c755421025a9e5165e03468a86d6de7397865be7187a63" Jul 24 00:14:49.815786 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:49.815739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:49.816254 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:49.815816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:49.816254 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:49.815847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnkk\" (UniqueName: \"kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk\") pod \"network-check-target-xnrdp\" (UID: \"84eade1e-4ce4-4c40-84f7-85ea4dfae29b\") " pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:49.818397 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:49.818365 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdnkk\" (UniqueName: \"kubernetes.io/projected/84eade1e-4ce4-4c40-84f7-85ea4dfae29b-kube-api-access-zdnkk\") pod \"network-check-target-xnrdp\" (UID: \"84eade1e-4ce4-4c40-84f7-85ea4dfae29b\") " pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:49.818397 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:49.818382 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 00:14:49.819218 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:49.819199 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 00:14:49.829224 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:49.829196 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6acafeb4-4bd5-4969-8722-6003504206b0-metrics-certs\") pod \"network-metrics-daemon-9dmnl\" (UID: \"6acafeb4-4bd5-4969-8722-6003504206b0\") " pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:49.829491 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:49.829469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c21f04e-dbb0-4743-831f-462ea9ffcd9c-original-pull-secret\") pod \"global-pull-secret-syncer-vncds\" (UID: \"3c21f04e-dbb0-4743-831f-462ea9ffcd9c\") " pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:50.004989 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:50.004957 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vncds" Jul 24 00:14:50.109054 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:50.109017 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jq7gh\"" Jul 24 00:14:50.112609 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:50.112590 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gpdt7\"" Jul 24 00:14:50.117317 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:50.117298 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dmnl" Jul 24 00:14:50.121521 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:50.121504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:50.663990 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:50.663903 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vncds"] Jul 24 00:14:50.692716 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:50.692686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9dmnl"] Jul 24 00:14:50.719007 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:50.718977 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xnrdp"] Jul 24 00:14:50.762686 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:50.762648 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c21f04e_dbb0_4743_831f_462ea9ffcd9c.slice/crio-c44e7f72bc80808814603e5c6863c41bb427083f2b14a02b1efc316a7d55ea39 WatchSource:0}: Error finding container c44e7f72bc80808814603e5c6863c41bb427083f2b14a02b1efc316a7d55ea39: Status 404 returned error can't find the container with id c44e7f72bc80808814603e5c6863c41bb427083f2b14a02b1efc316a7d55ea39 Jul 24 00:14:50.763519 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:50.763290 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acafeb4_4bd5_4969_8722_6003504206b0.slice/crio-afddbb3f48e3359a468984cb24c6f1404233e642dc0b0cea9c1f964364c6dc31 WatchSource:0}: Error finding container afddbb3f48e3359a468984cb24c6f1404233e642dc0b0cea9c1f964364c6dc31: Status 404 returned error can't find the container with id afddbb3f48e3359a468984cb24c6f1404233e642dc0b0cea9c1f964364c6dc31 Jul 24 00:14:50.764049 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:50.764026 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84eade1e_4ce4_4c40_84f7_85ea4dfae29b.slice/crio-84cf0be767f76f258f9abad0c03dfdf8701e56fb4e7c6a6f93737b99cdf42e4b WatchSource:0}: Error finding container 84cf0be767f76f258f9abad0c03dfdf8701e56fb4e7c6a6f93737b99cdf42e4b: Status 404 returned error can't find the container with id 84cf0be767f76f258f9abad0c03dfdf8701e56fb4e7c6a6f93737b99cdf42e4b Jul 24 00:14:51.227110 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.227078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:51.229689 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.229662 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03241e12-da01-4d42-abc0-30f73c4bb33b-metrics-tls\") pod \"dns-default-nw9wd\" (UID: \"03241e12-da01-4d42-abc0-30f73c4bb33b\") " pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:51.284921 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.284881 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9dmnl" event={"ID":"6acafeb4-4bd5-4969-8722-6003504206b0","Type":"ContainerStarted","Data":"afddbb3f48e3359a468984cb24c6f1404233e642dc0b0cea9c1f964364c6dc31"} Jul 24 00:14:51.287011 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.286976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k66cd" event={"ID":"04e9e66a-b4ae-4fe2-a876-440487e20e2b","Type":"ContainerStarted","Data":"390281a6101520f2cffba090cac5d26080d52ed9aa6e2fbe1a4ae21a83b96f83"} Jul 24 00:14:51.288470 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.288446 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xnrdp" event={"ID":"84eade1e-4ce4-4c40-84f7-85ea4dfae29b","Type":"ContainerStarted","Data":"dc64e3057b8fd41727f44850a4360e67b0f3243ef72b82375c8c39b26379ccd1"} Jul 24 00:14:51.288640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.288501 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xnrdp" event={"ID":"84eade1e-4ce4-4c40-84f7-85ea4dfae29b","Type":"ContainerStarted","Data":"84cf0be767f76f258f9abad0c03dfdf8701e56fb4e7c6a6f93737b99cdf42e4b"} Jul 24 00:14:51.288640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.288548 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:14:51.290669 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.290644 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:14:51.290791 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.290770 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" event={"ID":"6015e731-af19-441b-8bd1-19f859ed77bb","Type":"ContainerStarted","Data":"3794a268f4eb8dd58d7c7bd908e1c36b574c8cb52804f75572ef2ec654b323e7"} Jul 24 00:14:51.291272 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.291250 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:51.292883 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.292841 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vncds" event={"ID":"3c21f04e-dbb0-4743-831f-462ea9ffcd9c","Type":"ContainerStarted","Data":"c44e7f72bc80808814603e5c6863c41bb427083f2b14a02b1efc316a7d55ea39"} Jul 24 00:14:51.297319 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.297297 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" Jul 24 00:14:51.304054 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.303595 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xnrdp" podStartSLOduration=67.303577263 podStartE2EDuration="1m7.303577263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:51.302777976 +0000 UTC m=+67.875874778" watchObservedRunningTime="2026-07-24 00:14:51.303577263 +0000 UTC m=+67.876674066" Jul 24 00:14:51.424254 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.424215 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dvjcq\"" Jul 24 00:14:51.433807 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.433778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:51.610142 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.609465 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-r6gpd" podStartSLOduration=23.887950514 podStartE2EDuration="28.609442154s" totalImagesPullingTime="4.72149164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:51.318630763 +0000 UTC m=+67.891727567" watchObservedRunningTime="2026-07-24 00:14:51.609442154 +0000 UTC m=+68.182538963" Jul 24 00:14:51.610142 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:51.610039 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nw9wd"] Jul 24 00:14:51.968647 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:51.968608 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03241e12_da01_4d42_abc0_30f73c4bb33b.slice/crio-a900c42dd41bd0c6ac44a87fb68ae66dc0071f3dd1315ab6875e006bbc3ca326 WatchSource:0}: Error finding container a900c42dd41bd0c6ac44a87fb68ae66dc0071f3dd1315ab6875e006bbc3ca326: Status 404 returned error can't find the container with id a900c42dd41bd0c6ac44a87fb68ae66dc0071f3dd1315ab6875e006bbc3ca326 Jul 24 00:14:52.299102 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:52.299064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nw9wd" event={"ID":"03241e12-da01-4d42-abc0-30f73c4bb33b","Type":"ContainerStarted","Data":"a900c42dd41bd0c6ac44a87fb68ae66dc0071f3dd1315ab6875e006bbc3ca326"} Jul 24 00:14:53.309811 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.309701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9dmnl" event={"ID":"6acafeb4-4bd5-4969-8722-6003504206b0","Type":"ContainerStarted","Data":"2c26f5bb9cdbed23016c2bba2b0651d7a88a9f69117a76e4a2e8063ebc96f247"} Jul 24 00:14:53.309811 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.309761 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9dmnl" event={"ID":"6acafeb4-4bd5-4969-8722-6003504206b0","Type":"ContainerStarted","Data":"a8d7c95f0eb0dc9807f8f3ce984b06245b7a88b3fee97454c10809041b6582fe"} Jul 24 00:14:53.329209 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.328775 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9dmnl" podStartSLOduration=67.803442365 podStartE2EDuration="1m9.328754499s" totalImagesPullingTime="1.525312134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:53.326831885 +0000 UTC m=+69.899928691" watchObservedRunningTime="2026-07-24 00:14:53.328754499 +0000 UTC m=+69.901851304" Jul 24 00:14:53.405700 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.405647 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z"] Jul 24 00:14:53.407985 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.407961 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:14:53.410820 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.410793 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 24 00:14:53.411100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.411082 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 00:14:53.411281 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.411217 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 00:14:53.411408 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.411378 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 00:14:53.428206 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.428145 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z"] Jul 24 00:14:53.429540 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.429508 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c4d5c85d5-52lrm"] Jul 24 00:14:53.462017 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.461991 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cf7444644-zwzcq"] Jul 24 00:14:53.464595 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.464572 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.478337 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.478302 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cf7444644-zwzcq"] Jul 24 00:14:53.503679 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.503647 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-7sbfn"] Jul 24 00:14:53.505909 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.505883 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-7sbfn" Jul 24 00:14:53.508284 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.508262 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-t7gz9\"" Jul 24 00:14:53.508410 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.508294 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 00:14:53.508588 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.508571 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 00:14:53.516136 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.516113 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-7sbfn"] Jul 24 00:14:53.550684 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.550649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/818f9fad-8b07-4d09-80cd-6574438f43e1-klusterlet-config\") pod \"klusterlet-addon-workmgr-67c9578c55-n942z\" (UID: \"818f9fad-8b07-4d09-80cd-6574438f43e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:14:53.550827 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.550711 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sxhm\" (UniqueName: \"kubernetes.io/projected/818f9fad-8b07-4d09-80cd-6574438f43e1-kube-api-access-4sxhm\") pod \"klusterlet-addon-workmgr-67c9578c55-n942z\" (UID: \"818f9fad-8b07-4d09-80cd-6574438f43e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:14:53.550827 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.550762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/459d4439-0f5a-4a4e-a30b-0918f09496d0-image-registry-private-configuration\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.550827 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.550805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/818f9fad-8b07-4d09-80cd-6574438f43e1-tmp\") pod \"klusterlet-addon-workmgr-67c9578c55-n942z\" (UID: \"818f9fad-8b07-4d09-80cd-6574438f43e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:14:53.551007 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.550858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/459d4439-0f5a-4a4e-a30b-0918f09496d0-trusted-ca\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.551007 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.550889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/459d4439-0f5a-4a4e-a30b-0918f09496d0-bound-sa-token\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.551007 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.550967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/459d4439-0f5a-4a4e-a30b-0918f09496d0-registry-tls\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.551140 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.551005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/459d4439-0f5a-4a4e-a30b-0918f09496d0-ca-trust-extracted\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.551140 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.551038 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/459d4439-0f5a-4a4e-a30b-0918f09496d0-registry-certificates\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.551140 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.551124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/459d4439-0f5a-4a4e-a30b-0918f09496d0-installation-pull-secrets\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.551351 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.551176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4zsc\" (UniqueName: \"kubernetes.io/projected/459d4439-0f5a-4a4e-a30b-0918f09496d0-kube-api-access-g4zsc\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.652499 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.652412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/459d4439-0f5a-4a4e-a30b-0918f09496d0-trusted-ca\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.652669 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.652543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/459d4439-0f5a-4a4e-a30b-0918f09496d0-bound-sa-token\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.652669 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.652585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/459d4439-0f5a-4a4e-a30b-0918f09496d0-registry-tls\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.652669 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.652605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xszk\" (UniqueName: \"kubernetes.io/projected/a36a2229-e2b1-4935-81ff-b2ff0de97247-kube-api-access-8xszk\") pod \"downloads-7c97dc4555-7sbfn\" (UID: \"a36a2229-e2b1-4935-81ff-b2ff0de97247\") " pod="openshift-console/downloads-7c97dc4555-7sbfn" Jul 24 00:14:53.652669 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.652624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/459d4439-0f5a-4a4e-a30b-0918f09496d0-ca-trust-extracted\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.652669 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.652648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/459d4439-0f5a-4a4e-a30b-0918f09496d0-registry-certificates\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.652935 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.652854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/459d4439-0f5a-4a4e-a30b-0918f09496d0-installation-pull-secrets\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.652935 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.652893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4zsc\" (UniqueName: \"kubernetes.io/projected/459d4439-0f5a-4a4e-a30b-0918f09496d0-kube-api-access-g4zsc\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.653053 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.652937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/818f9fad-8b07-4d09-80cd-6574438f43e1-klusterlet-config\") pod \"klusterlet-addon-workmgr-67c9578c55-n942z\" (UID: \"818f9fad-8b07-4d09-80cd-6574438f43e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:14:53.653053 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.652977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sxhm\" (UniqueName: \"kubernetes.io/projected/818f9fad-8b07-4d09-80cd-6574438f43e1-kube-api-access-4sxhm\") pod \"klusterlet-addon-workmgr-67c9578c55-n942z\" (UID: \"818f9fad-8b07-4d09-80cd-6574438f43e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:14:53.653053 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.653004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/459d4439-0f5a-4a4e-a30b-0918f09496d0-image-registry-private-configuration\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.653053 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.653036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/818f9fad-8b07-4d09-80cd-6574438f43e1-tmp\") pod \"klusterlet-addon-workmgr-67c9578c55-n942z\" (UID: \"818f9fad-8b07-4d09-80cd-6574438f43e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:14:53.653248 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.653047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/459d4439-0f5a-4a4e-a30b-0918f09496d0-ca-trust-extracted\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.653654 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.653412 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/818f9fad-8b07-4d09-80cd-6574438f43e1-tmp\") pod \"klusterlet-addon-workmgr-67c9578c55-n942z\" (UID: \"818f9fad-8b07-4d09-80cd-6574438f43e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:14:53.653654 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.653512 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/459d4439-0f5a-4a4e-a30b-0918f09496d0-trusted-ca\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.653654 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.653605 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/459d4439-0f5a-4a4e-a30b-0918f09496d0-registry-certificates\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.656001 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.655942 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/459d4439-0f5a-4a4e-a30b-0918f09496d0-registry-tls\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.656162 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.656139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/459d4439-0f5a-4a4e-a30b-0918f09496d0-installation-pull-secrets\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.656591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.656564 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/459d4439-0f5a-4a4e-a30b-0918f09496d0-image-registry-private-configuration\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.657350 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.657327 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/818f9fad-8b07-4d09-80cd-6574438f43e1-klusterlet-config\") pod \"klusterlet-addon-workmgr-67c9578c55-n942z\" (UID: \"818f9fad-8b07-4d09-80cd-6574438f43e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:14:53.660912 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.660883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/459d4439-0f5a-4a4e-a30b-0918f09496d0-bound-sa-token\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.661340 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.661260 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4zsc\" (UniqueName: \"kubernetes.io/projected/459d4439-0f5a-4a4e-a30b-0918f09496d0-kube-api-access-g4zsc\") pod \"image-registry-cf7444644-zwzcq\" (UID: \"459d4439-0f5a-4a4e-a30b-0918f09496d0\") " pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.662100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.662075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sxhm\" (UniqueName: \"kubernetes.io/projected/818f9fad-8b07-4d09-80cd-6574438f43e1-kube-api-access-4sxhm\") pod \"klusterlet-addon-workmgr-67c9578c55-n942z\" (UID: \"818f9fad-8b07-4d09-80cd-6574438f43e1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:14:53.719782 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.719741 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:14:53.753786 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.753717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xszk\" (UniqueName: \"kubernetes.io/projected/a36a2229-e2b1-4935-81ff-b2ff0de97247-kube-api-access-8xszk\") pod \"downloads-7c97dc4555-7sbfn\" (UID: \"a36a2229-e2b1-4935-81ff-b2ff0de97247\") " pod="openshift-console/downloads-7c97dc4555-7sbfn" Jul 24 00:14:53.768253 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.768216 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xszk\" (UniqueName: \"kubernetes.io/projected/a36a2229-e2b1-4935-81ff-b2ff0de97247-kube-api-access-8xszk\") pod \"downloads-7c97dc4555-7sbfn\" (UID: \"a36a2229-e2b1-4935-81ff-b2ff0de97247\") " pod="openshift-console/downloads-7c97dc4555-7sbfn" Jul 24 00:14:53.775057 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.775025 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:53.818175 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:53.818134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-7sbfn" Jul 24 00:14:55.672352 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:55.672312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-stcch\" (UID: \"dabcdd63-5694-4208-826b-6682b43c826d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:55.674920 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:55.674882 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dabcdd63-5694-4208-826b-6682b43c826d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-stcch\" (UID: \"dabcdd63-5694-4208-826b-6682b43c826d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:55.804722 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:55.804692 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rwv72\"" Jul 24 00:14:55.812649 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:55.812611 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" Jul 24 00:14:57.000206 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.000139 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z"] Jul 24 00:14:57.004583 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:57.004434 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818f9fad_8b07_4d09_80cd_6574438f43e1.slice/crio-5d22a21e6898b432962bd099601adb3e8f3b02f9e2109e54b7344f743277b0bd WatchSource:0}: Error finding container 5d22a21e6898b432962bd099601adb3e8f3b02f9e2109e54b7344f743277b0bd: Status 404 returned error can't find the container with id 5d22a21e6898b432962bd099601adb3e8f3b02f9e2109e54b7344f743277b0bd Jul 24 00:14:57.024636 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.024569 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cf7444644-zwzcq"] Jul 24 00:14:57.262455 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.262325 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-7sbfn"] Jul 24 00:14:57.264371 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.264345 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-stcch"] Jul 24 00:14:57.266916 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:57.266890 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36a2229_e2b1_4935_81ff_b2ff0de97247.slice/crio-84038f4ba42d2ee830579a106487e41e7a791b1f58ce1c2b4b2cc7a6ecb57eef WatchSource:0}: Error finding container 84038f4ba42d2ee830579a106487e41e7a791b1f58ce1c2b4b2cc7a6ecb57eef: Status 404 returned error can't find the container with id 84038f4ba42d2ee830579a106487e41e7a791b1f58ce1c2b4b2cc7a6ecb57eef Jul 24 00:14:57.267925 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:14:57.267902 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabcdd63_5694_4208_826b_6682b43c826d.slice/crio-083d24c2809ee1a00b9f22c241738f284da32369a985720a3aa149ff7822b89f WatchSource:0}: Error finding container 083d24c2809ee1a00b9f22c241738f284da32369a985720a3aa149ff7822b89f: Status 404 returned error can't find the container with id 083d24c2809ee1a00b9f22c241738f284da32369a985720a3aa149ff7822b89f Jul 24 00:14:57.325456 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.325402 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cf7444644-zwzcq" event={"ID":"459d4439-0f5a-4a4e-a30b-0918f09496d0","Type":"ContainerStarted","Data":"371e7ac1bc64397f9fbf87a8d8a064e0ac84da733831e12efcf30aa81d7b4c99"} Jul 24 00:14:57.325650 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.325462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cf7444644-zwzcq" event={"ID":"459d4439-0f5a-4a4e-a30b-0918f09496d0","Type":"ContainerStarted","Data":"667657d6677c49c9fdc86ce596d0a6b7442e10bba5dcd2eb34264d8d1f262c5b"} Jul 24 00:14:57.325650 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.325623 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:14:57.326932 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.326897 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vncds" event={"ID":"3c21f04e-dbb0-4743-831f-462ea9ffcd9c","Type":"ContainerStarted","Data":"e113a5de0d20eb270e728c4b5f30d1c7a9c7b0a5035dc88820694a3cc48e6341"} Jul 24 00:14:57.328681 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.328658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k66cd" event={"ID":"04e9e66a-b4ae-4fe2-a876-440487e20e2b","Type":"ContainerStarted","Data":"ea11a843287ba33d48e9783ff861b92e97bb532ab4c9bb7666403f0932281232"} Jul 24 00:14:57.329752 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.329733 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" event={"ID":"dabcdd63-5694-4208-826b-6682b43c826d","Type":"ContainerStarted","Data":"083d24c2809ee1a00b9f22c241738f284da32369a985720a3aa149ff7822b89f"} Jul 24 00:14:57.330843 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.330819 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" event={"ID":"818f9fad-8b07-4d09-80cd-6574438f43e1","Type":"ContainerStarted","Data":"5d22a21e6898b432962bd099601adb3e8f3b02f9e2109e54b7344f743277b0bd"} Jul 24 00:14:57.332551 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.332532 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nw9wd" event={"ID":"03241e12-da01-4d42-abc0-30f73c4bb33b","Type":"ContainerStarted","Data":"fdacc8b5095cebe090d4d472317eb1596c21bd15af0e3afd9e33e3bb5e643b10"} Jul 24 00:14:57.332647 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.332574 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nw9wd" event={"ID":"03241e12-da01-4d42-abc0-30f73c4bb33b","Type":"ContainerStarted","Data":"68e692700713ebbf7905483e482c606f59995e72d628cea750ba03eda8472a17"} Jul 24 00:14:57.332647 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.332628 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nw9wd" Jul 24 00:14:57.333596 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.333580 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-7sbfn" event={"ID":"a36a2229-e2b1-4935-81ff-b2ff0de97247","Type":"ContainerStarted","Data":"84038f4ba42d2ee830579a106487e41e7a791b1f58ce1c2b4b2cc7a6ecb57eef"} Jul 24 00:14:57.347655 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.347615 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cf7444644-zwzcq" podStartSLOduration=4.347603063 podStartE2EDuration="4.347603063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:57.346425699 +0000 UTC m=+73.919522501" watchObservedRunningTime="2026-07-24 00:14:57.347603063 +0000 UTC m=+73.920699866" Jul 24 00:14:57.360008 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.359969 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vncds" podStartSLOduration=67.301823561 podStartE2EDuration="1m13.359933101s" totalImagesPullingTime="6.05810954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:57.359554474 +0000 UTC m=+73.932651278" watchObservedRunningTime="2026-07-24 00:14:57.359933101 +0000 UTC m=+73.933029952" Jul 24 00:14:57.374990 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:14:57.374927 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nw9wd" podStartSLOduration=33.532815121 podStartE2EDuration="38.3748965s" totalImagesPullingTime="4.842081379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:57.374761478 +0000 UTC m=+73.947858280" watchObservedRunningTime="2026-07-24 00:14:57.3748965 +0000 UTC m=+73.947993298" Jul 24 00:15:03.436766 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:03.436513 2576 patch_prober.go:28] interesting pod/image-registry-5c4d5c85d5-52lrm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 00:15:03.436766 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:03.436581 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" podUID="9ae9d4a5-4472-4cc3-b83c-626d061d2b55" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 00:15:04.369030 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:04.368990 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k66cd" event={"ID":"04e9e66a-b4ae-4fe2-a876-440487e20e2b","Type":"ContainerStarted","Data":"6a53df9169e69feefb0f30f6c5d8c484185a019475ba7cb6287f1d9020c6cb22"} Jul 24 00:15:04.370634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:04.370605 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" event={"ID":"dabcdd63-5694-4208-826b-6682b43c826d","Type":"ContainerStarted","Data":"ccfd7a75b05105933d5a87d421b8bcc60f2dc4fe8989e55e86ea52d6d69c9e9e"} Jul 24 00:15:04.376417 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:04.376386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" event={"ID":"818f9fad-8b07-4d09-80cd-6574438f43e1","Type":"ContainerStarted","Data":"7febfbe345d4b6948afc1975315fb0296accfa113b50055a1eb50dde87c44577"} Jul 24 00:15:04.376535 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:04.376463 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:15:04.378429 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:04.378407 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" Jul 24 00:15:04.387585 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:04.387535 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k66cd" podStartSLOduration=17.389819325 podStartE2EDuration="35.387519338s" totalImagesPullingTime="17.997700013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:14:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:15:04.386145398 +0000 UTC m=+80.959242225" watchObservedRunningTime="2026-07-24 00:15:04.387519338 +0000 UTC m=+80.960616123" Jul 24 00:15:04.400926 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:04.400811 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-stcch" podStartSLOduration=34.756045851 podStartE2EDuration="41.400795393s" totalImagesPullingTime="6.644749542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:15:04.400666462 +0000 UTC m=+80.973763287" watchObservedRunningTime="2026-07-24 00:15:04.400795393 +0000 UTC m=+80.973892194" Jul 24 00:15:04.421983 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:04.421914 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9578c55-n942z" podStartSLOduration=4.487843065 podStartE2EDuration="11.421896199s" totalImagesPullingTime="6.934053134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:15:04.420715251 +0000 UTC m=+80.993812080" watchObservedRunningTime="2026-07-24 00:15:04.421896199 +0000 UTC m=+80.994993002" Jul 24 00:15:07.344444 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:07.344311 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nw9wd" Jul 24 00:15:10.898855 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.898813 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zkhtq"] Jul 24 00:15:10.910076 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.910044 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:10.912734 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.912708 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 00:15:10.912873 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.912738 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 00:15:10.912873 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.912765 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 00:15:10.913075 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.913056 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rtkhg\"" Jul 24 00:15:10.913887 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.913629 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 00:15:10.913887 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.913770 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 00:15:10.913887 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.913815 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 00:15:10.996364 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.996325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/63a819a0-6c03-4aec-9c03-4fa662e3c100-root\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:10.996557 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.996397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-accelerators-collector-config\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:10.996557 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.996450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bzvb\" (UniqueName: \"kubernetes.io/projected/63a819a0-6c03-4aec-9c03-4fa662e3c100-kube-api-access-9bzvb\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:10.996557 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.996478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63a819a0-6c03-4aec-9c03-4fa662e3c100-sys\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:10.996557 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.996502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63a819a0-6c03-4aec-9c03-4fa662e3c100-metrics-client-ca\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:10.996557 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.996532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-tls\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:10.996762 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.996562 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-textfile\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:10.996762 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.996587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-wtmp\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:10.996762 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:10.996610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.097699 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.097666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/63a819a0-6c03-4aec-9c03-4fa662e3c100-root\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.097884 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.097714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-accelerators-collector-config\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.097884 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.097749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bzvb\" (UniqueName: \"kubernetes.io/projected/63a819a0-6c03-4aec-9c03-4fa662e3c100-kube-api-access-9bzvb\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.097884 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.097778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63a819a0-6c03-4aec-9c03-4fa662e3c100-sys\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.097884 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.097792 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/63a819a0-6c03-4aec-9c03-4fa662e3c100-root\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.097884 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.097801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63a819a0-6c03-4aec-9c03-4fa662e3c100-metrics-client-ca\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.098189 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.097882 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63a819a0-6c03-4aec-9c03-4fa662e3c100-sys\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.098189 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.097972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-tls\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.098189 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.098017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-textfile\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.098189 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.098046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-wtmp\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.098189 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.098074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.098433 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.098409 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-accelerators-collector-config\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.098525 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.098497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-textfile\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.098653 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.098537 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-wtmp\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.098653 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.098540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63a819a0-6c03-4aec-9c03-4fa662e3c100-metrics-client-ca\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.101255 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.101231 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.101414 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.101290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/63a819a0-6c03-4aec-9c03-4fa662e3c100-node-exporter-tls\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.106369 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.106347 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bzvb\" (UniqueName: \"kubernetes.io/projected/63a819a0-6c03-4aec-9c03-4fa662e3c100-kube-api-access-9bzvb\") pod \"node-exporter-zkhtq\" (UID: \"63a819a0-6c03-4aec-9c03-4fa662e3c100\") " pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:11.220904 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:11.220824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zkhtq" Jul 24 00:15:13.434934 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:13.434907 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:15:15.029910 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:15:15.029874 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a819a0_6c03_4aec_9c03_4fa662e3c100.slice/crio-798aca08e63532277b3baf7748d542e9aa4a0d4da80bda35eb4e2e27d28bdc91 WatchSource:0}: Error finding container 798aca08e63532277b3baf7748d542e9aa4a0d4da80bda35eb4e2e27d28bdc91: Status 404 returned error can't find the container with id 798aca08e63532277b3baf7748d542e9aa4a0d4da80bda35eb4e2e27d28bdc91 Jul 24 00:15:15.419847 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:15.419804 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkhtq" event={"ID":"63a819a0-6c03-4aec-9c03-4fa662e3c100","Type":"ContainerStarted","Data":"798aca08e63532277b3baf7748d542e9aa4a0d4da80bda35eb4e2e27d28bdc91"} Jul 24 00:15:16.426606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:16.426518 2576 generic.go:356] "Generic (PLEG): container finished" podID="63a819a0-6c03-4aec-9c03-4fa662e3c100" containerID="4a054182b3b1b246782ca4008d4c0fe999b6413006208e8965991b736514c2c9" exitCode=0 Jul 24 00:15:16.427073 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:16.426611 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkhtq" event={"ID":"63a819a0-6c03-4aec-9c03-4fa662e3c100","Type":"ContainerDied","Data":"4a054182b3b1b246782ca4008d4c0fe999b6413006208e8965991b736514c2c9"} Jul 24 00:15:16.428582 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:16.428557 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-7sbfn" event={"ID":"a36a2229-e2b1-4935-81ff-b2ff0de97247","Type":"ContainerStarted","Data":"782e970b394e3e7e9110cab8adc39d4ae89f8786fe1b8b58f72e0a3093c8023a"} Jul 24 00:15:16.428764 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:16.428747 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-7sbfn" Jul 24 00:15:16.450320 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:16.450287 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-7sbfn" Jul 24 00:15:16.462494 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:16.462433 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-7sbfn" podStartSLOduration=5.282236952 podStartE2EDuration="23.462413472s" totalImagesPullingTime="18.18017652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:15:16.460182917 +0000 UTC m=+93.033279721" watchObservedRunningTime="2026-07-24 00:15:16.462413472 +0000 UTC m=+93.035510275" Jul 24 00:15:17.436217 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:17.436169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkhtq" event={"ID":"63a819a0-6c03-4aec-9c03-4fa662e3c100","Type":"ContainerStarted","Data":"15ebaa8bc5c60f10725e43bdd4e0925e0595d652666f7fca3c0b8d64e50d9bde"} Jul 24 00:15:17.436714 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:17.436289 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkhtq" event={"ID":"63a819a0-6c03-4aec-9c03-4fa662e3c100","Type":"ContainerStarted","Data":"02c1a06990bd05797d08ffa05e0a06ae18cf81a8a7ebda2007d37f0807f51e92"} Jul 24 00:15:17.455992 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:17.455907 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zkhtq" podStartSLOduration=6.353823936 podStartE2EDuration="7.455890298s" totalImagesPullingTime="1.01886302s" totalInitContainerRuntime="83.203342ms" isStatefulPod=true podCreationTimestamp="2026-07-24 00:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:15:17.455253874 +0000 UTC m=+94.028350676" watchObservedRunningTime="2026-07-24 00:15:17.455890298 +0000 UTC m=+94.028987100" Jul 24 00:15:18.347032 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.346996 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cf7444644-zwzcq" Jul 24 00:15:18.458914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.458877 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" podUID="9ae9d4a5-4472-4cc3-b83c-626d061d2b55" containerName="registry" containerID="cri-o://0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875" gracePeriod=30 Jul 24 00:15:18.769900 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.769862 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:15:18.862708 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.862627 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5tsx\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-kube-api-access-w5tsx\") pod \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " Jul 24 00:15:18.862708 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.862673 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-certificates\") pod \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " Jul 24 00:15:18.862708 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.862698 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-bound-sa-token\") pod \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " Jul 24 00:15:18.863022 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.862734 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-image-registry-private-configuration\") pod \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " Jul 24 00:15:18.863022 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.862769 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-trusted-ca\") pod \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " Jul 24 00:15:18.863022 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.862815 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls\") pod \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " Jul 24 00:15:18.863022 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.862869 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-ca-trust-extracted\") pod \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " Jul 24 00:15:18.863022 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.862902 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-installation-pull-secrets\") pod \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\" (UID: \"9ae9d4a5-4472-4cc3-b83c-626d061d2b55\") " Jul 24 00:15:18.863299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.863058 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9ae9d4a5-4472-4cc3-b83c-626d061d2b55" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:15:18.863299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.863182 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-certificates\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:15:18.866308 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.866252 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9ae9d4a5-4472-4cc3-b83c-626d061d2b55" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:15:18.866557 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.866515 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-kube-api-access-w5tsx" (OuterVolumeSpecName: "kube-api-access-w5tsx") pod "9ae9d4a5-4472-4cc3-b83c-626d061d2b55" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55"). InnerVolumeSpecName "kube-api-access-w5tsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:15:18.866764 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.866737 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9ae9d4a5-4472-4cc3-b83c-626d061d2b55" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:15:18.866888 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.866869 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9ae9d4a5-4472-4cc3-b83c-626d061d2b55" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:15:18.867875 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.867843 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9ae9d4a5-4472-4cc3-b83c-626d061d2b55" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:15:18.868038 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.868015 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9ae9d4a5-4472-4cc3-b83c-626d061d2b55" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:15:18.874518 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.874478 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9ae9d4a5-4472-4cc3-b83c-626d061d2b55" (UID: "9ae9d4a5-4472-4cc3-b83c-626d061d2b55"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:15:18.964615 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.964575 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5tsx\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-kube-api-access-w5tsx\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:15:18.964615 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.964613 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-bound-sa-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:15:18.964837 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.964628 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-image-registry-private-configuration\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:15:18.964837 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.964639 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-trusted-ca\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:15:18.964837 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.964648 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-registry-tls\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:15:18.964837 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.964655 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-ca-trust-extracted\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:15:18.964837 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:18.964663 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ae9d4a5-4472-4cc3-b83c-626d061d2b55-installation-pull-secrets\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:15:19.445334 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:19.445298 2576 generic.go:356] "Generic (PLEG): container finished" podID="9ae9d4a5-4472-4cc3-b83c-626d061d2b55" containerID="0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875" exitCode=0 Jul 24 00:15:19.445595 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:19.445350 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" Jul 24 00:15:19.445595 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:19.445385 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" event={"ID":"9ae9d4a5-4472-4cc3-b83c-626d061d2b55","Type":"ContainerDied","Data":"0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875"} Jul 24 00:15:19.445595 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:19.445445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c4d5c85d5-52lrm" event={"ID":"9ae9d4a5-4472-4cc3-b83c-626d061d2b55","Type":"ContainerDied","Data":"58bc06fc1953cbbb36bde4ffa12d13ad42d4e272f44ad3af96fb706e1642406c"} Jul 24 00:15:19.445595 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:19.445473 2576 scope.go:117] "RemoveContainer" containerID="0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875" Jul 24 00:15:19.455224 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:19.455168 2576 scope.go:117] "RemoveContainer" containerID="0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875" Jul 24 00:15:19.455692 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:15:19.455658 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875\": container with ID starting with 0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875 not found: ID does not exist" containerID="0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875" Jul 24 00:15:19.455785 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:19.455707 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875"} err="failed to get container status \"0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875\": rpc error: code = NotFound desc = could not find container \"0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875\": container with ID starting with 0ef8b23a37fb5fbbf53d9a2077ec436dc9ffab2ad261bb3b0a3a0eeca80c3875 not found: ID does not exist" Jul 24 00:15:19.470140 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:19.470103 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c4d5c85d5-52lrm"] Jul 24 00:15:19.473601 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:19.473573 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c4d5c85d5-52lrm"] Jul 24 00:15:20.007235 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:20.007176 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae9d4a5-4472-4cc3-b83c-626d061d2b55" path="/var/lib/kubelet/pods/9ae9d4a5-4472-4cc3-b83c-626d061d2b55/volumes" Jul 24 00:15:22.302147 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:22.302116 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xnrdp" Jul 24 00:15:59.592173 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:59.592140 2576 generic.go:356] "Generic (PLEG): container finished" podID="86ed4490-c0cd-45a0-bfe0-ba5730e4b621" containerID="60da693b3be2fa1bf47cce9b1513cdda251ed47476797d97a4c4c7a473d82fd1" exitCode=0 Jul 24 00:15:59.592567 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:59.592215 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-pnv5b" event={"ID":"86ed4490-c0cd-45a0-bfe0-ba5730e4b621","Type":"ContainerDied","Data":"60da693b3be2fa1bf47cce9b1513cdda251ed47476797d97a4c4c7a473d82fd1"} Jul 24 00:15:59.592567 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:15:59.592547 2576 scope.go:117] "RemoveContainer" containerID="60da693b3be2fa1bf47cce9b1513cdda251ed47476797d97a4c4c7a473d82fd1" Jul 24 00:16:00.597370 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:00.597332 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-pnv5b" event={"ID":"86ed4490-c0cd-45a0-bfe0-ba5730e4b621","Type":"ContainerStarted","Data":"c2413b452135df7f5e1881e775db3442299a2c0da2909db8736e0ff9e7021d20"} Jul 24 00:16:35.179293 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.179255 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7896f5b49c-h8wnl"] Jul 24 00:16:35.179778 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.179586 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae9d4a5-4472-4cc3-b83c-626d061d2b55" containerName="registry" Jul 24 00:16:35.179778 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.179605 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae9d4a5-4472-4cc3-b83c-626d061d2b55" containerName="registry" Jul 24 00:16:35.179778 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.179667 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae9d4a5-4472-4cc3-b83c-626d061d2b55" containerName="registry" Jul 24 00:16:35.181817 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.181799 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.193279 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.193256 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 24 00:16:35.193279 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.193269 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 24 00:16:35.193470 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.193325 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 24 00:16:35.193542 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.193529 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 24 00:16:35.194296 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.194278 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 24 00:16:35.194414 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.194396 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wcksl\"" Jul 24 00:16:35.201285 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.201266 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 24 00:16:35.214218 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.214191 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7896f5b49c-h8wnl"] Jul 24 00:16:35.241808 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.241770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d15a62c-f665-4abb-ad18-e97811f54a3b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.241808 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.241810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrd2\" (UniqueName: \"kubernetes.io/projected/3d15a62c-f665-4abb-ad18-e97811f54a3b-kube-api-access-xjrd2\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.242038 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.241832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d15a62c-f665-4abb-ad18-e97811f54a3b-serving-certs-ca-bundle\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.242038 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.241882 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3d15a62c-f665-4abb-ad18-e97811f54a3b-telemeter-client-tls\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.242038 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.241941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3d15a62c-f665-4abb-ad18-e97811f54a3b-secret-telemeter-client\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.242038 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.241996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d15a62c-f665-4abb-ad18-e97811f54a3b-metrics-client-ca\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.242038 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.242032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d15a62c-f665-4abb-ad18-e97811f54a3b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.242197 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.242076 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3d15a62c-f665-4abb-ad18-e97811f54a3b-federate-client-tls\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.343456 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.343407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3d15a62c-f665-4abb-ad18-e97811f54a3b-federate-client-tls\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.343456 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.343463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d15a62c-f665-4abb-ad18-e97811f54a3b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.343750 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.343719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrd2\" (UniqueName: \"kubernetes.io/projected/3d15a62c-f665-4abb-ad18-e97811f54a3b-kube-api-access-xjrd2\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.343816 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.343769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d15a62c-f665-4abb-ad18-e97811f54a3b-serving-certs-ca-bundle\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.343816 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.343797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3d15a62c-f665-4abb-ad18-e97811f54a3b-telemeter-client-tls\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.344000 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.343964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3d15a62c-f665-4abb-ad18-e97811f54a3b-secret-telemeter-client\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.344127 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.344025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d15a62c-f665-4abb-ad18-e97811f54a3b-metrics-client-ca\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.344127 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.344064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d15a62c-f665-4abb-ad18-e97811f54a3b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.344556 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.344529 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d15a62c-f665-4abb-ad18-e97811f54a3b-serving-certs-ca-bundle\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.344687 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.344663 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d15a62c-f665-4abb-ad18-e97811f54a3b-metrics-client-ca\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.344779 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.344566 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d15a62c-f665-4abb-ad18-e97811f54a3b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.346263 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.346241 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3d15a62c-f665-4abb-ad18-e97811f54a3b-federate-client-tls\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.346348 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.346262 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3d15a62c-f665-4abb-ad18-e97811f54a3b-telemeter-client-tls\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.346573 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.346554 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d15a62c-f665-4abb-ad18-e97811f54a3b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.346682 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.346663 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3d15a62c-f665-4abb-ad18-e97811f54a3b-secret-telemeter-client\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.354298 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.354271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrd2\" (UniqueName: \"kubernetes.io/projected/3d15a62c-f665-4abb-ad18-e97811f54a3b-kube-api-access-xjrd2\") pod \"telemeter-client-7896f5b49c-h8wnl\" (UID: \"3d15a62c-f665-4abb-ad18-e97811f54a3b\") " pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.491171 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.491083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" Jul 24 00:16:35.630804 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.630774 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7896f5b49c-h8wnl"] Jul 24 00:16:35.634025 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:16:35.633998 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d15a62c_f665_4abb_ad18_e97811f54a3b.slice/crio-a2e741333fe25b767f6a2098669976d57c7ebbcb9485888d61c536d3b21db762 WatchSource:0}: Error finding container a2e741333fe25b767f6a2098669976d57c7ebbcb9485888d61c536d3b21db762: Status 404 returned error can't find the container with id a2e741333fe25b767f6a2098669976d57c7ebbcb9485888d61c536d3b21db762 Jul 24 00:16:35.719515 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:35.719476 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" event={"ID":"3d15a62c-f665-4abb-ad18-e97811f54a3b","Type":"ContainerStarted","Data":"a2e741333fe25b767f6a2098669976d57c7ebbcb9485888d61c536d3b21db762"} Jul 24 00:16:37.728164 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:37.728128 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" event={"ID":"3d15a62c-f665-4abb-ad18-e97811f54a3b","Type":"ContainerStarted","Data":"6dbd2e57f772d232aaa45cb3b8fe374e21f062b3a9d8ecd6bc65054152e95c11"} Jul 24 00:16:39.737811 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:39.737773 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" event={"ID":"3d15a62c-f665-4abb-ad18-e97811f54a3b","Type":"ContainerStarted","Data":"85af1e57357e11933c2bf3e8825b8c90ae36175572cc1d4b889dbc4ab41de0de"} Jul 24 00:16:39.738220 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:39.737867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" event={"ID":"3d15a62c-f665-4abb-ad18-e97811f54a3b","Type":"ContainerStarted","Data":"e43a2529066f74202e2024165614c0a077b86573d45c3a05c26daedaf26bfd94"} Jul 24 00:16:39.760059 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:16:39.760006 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7896f5b49c-h8wnl" podStartSLOduration=1.688447659 podStartE2EDuration="4.759988241s" totalImagesPullingTime="3.071540582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:16:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:16:39.757934067 +0000 UTC m=+176.331030910" watchObservedRunningTime="2026-07-24 00:16:39.759988241 +0000 UTC m=+176.333085045" Jul 24 00:17:49.731170 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.728769 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4"] Jul 24 00:17:49.733268 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.733239 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:17:49.736195 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.736170 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 00:17:49.736370 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.736353 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 00:17:49.737231 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.737217 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-j9flg\"" Jul 24 00:17:49.743852 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.743826 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4"] Jul 24 00:17:49.842494 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.842447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07aa5b38-7934-4ff6-baff-0bdf18326138-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4\" (UID: \"07aa5b38-7934-4ff6-baff-0bdf18326138\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:17:49.842685 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.842508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07aa5b38-7934-4ff6-baff-0bdf18326138-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4\" (UID: \"07aa5b38-7934-4ff6-baff-0bdf18326138\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:17:49.842685 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.842546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5scl\" (UniqueName: \"kubernetes.io/projected/07aa5b38-7934-4ff6-baff-0bdf18326138-kube-api-access-r5scl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4\" (UID: \"07aa5b38-7934-4ff6-baff-0bdf18326138\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:17:49.943562 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.943518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07aa5b38-7934-4ff6-baff-0bdf18326138-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4\" (UID: \"07aa5b38-7934-4ff6-baff-0bdf18326138\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:17:49.943562 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.943568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5scl\" (UniqueName: \"kubernetes.io/projected/07aa5b38-7934-4ff6-baff-0bdf18326138-kube-api-access-r5scl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4\" (UID: \"07aa5b38-7934-4ff6-baff-0bdf18326138\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:17:49.943827 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.943618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07aa5b38-7934-4ff6-baff-0bdf18326138-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4\" (UID: \"07aa5b38-7934-4ff6-baff-0bdf18326138\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:17:49.944018 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.943994 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07aa5b38-7934-4ff6-baff-0bdf18326138-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4\" (UID: \"07aa5b38-7934-4ff6-baff-0bdf18326138\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:17:49.944093 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.944034 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07aa5b38-7934-4ff6-baff-0bdf18326138-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4\" (UID: \"07aa5b38-7934-4ff6-baff-0bdf18326138\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:17:49.953468 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:49.953442 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5scl\" (UniqueName: \"kubernetes.io/projected/07aa5b38-7934-4ff6-baff-0bdf18326138-kube-api-access-r5scl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4\" (UID: \"07aa5b38-7934-4ff6-baff-0bdf18326138\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:17:50.042256 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:50.042226 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:17:50.173373 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:50.173339 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4"] Jul 24 00:17:50.176886 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:17:50.176854 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07aa5b38_7934_4ff6_baff_0bdf18326138.slice/crio-272d7a6ed40e4101fb480141db0a7634989e79832bb52ab48fed2e48ee8c4325 WatchSource:0}: Error finding container 272d7a6ed40e4101fb480141db0a7634989e79832bb52ab48fed2e48ee8c4325: Status 404 returned error can't find the container with id 272d7a6ed40e4101fb480141db0a7634989e79832bb52ab48fed2e48ee8c4325 Jul 24 00:17:50.991518 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:50.991478 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" event={"ID":"07aa5b38-7934-4ff6-baff-0bdf18326138","Type":"ContainerStarted","Data":"272d7a6ed40e4101fb480141db0a7634989e79832bb52ab48fed2e48ee8c4325"} Jul 24 00:17:56.009862 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:56.009824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" event={"ID":"07aa5b38-7934-4ff6-baff-0bdf18326138","Type":"ContainerStarted","Data":"b3ae812f1796ec9b5722db0ea315db6628768f3db45520d535e6ba3439577b94"} Jul 24 00:17:57.015207 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:57.015172 2576 generic.go:356] "Generic (PLEG): container finished" podID="07aa5b38-7934-4ff6-baff-0bdf18326138" containerID="b3ae812f1796ec9b5722db0ea315db6628768f3db45520d535e6ba3439577b94" exitCode=0 Jul 24 00:17:57.015610 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:17:57.015272 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" event={"ID":"07aa5b38-7934-4ff6-baff-0bdf18326138","Type":"ContainerDied","Data":"b3ae812f1796ec9b5722db0ea315db6628768f3db45520d535e6ba3439577b94"} Jul 24 00:18:00.027160 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:00.027124 2576 generic.go:356] "Generic (PLEG): container finished" podID="07aa5b38-7934-4ff6-baff-0bdf18326138" containerID="9c5fab005e67237e5d205d153a50227e55cd722b2e1ae9379cebee9d9542d4e5" exitCode=0 Jul 24 00:18:00.027551 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:00.027179 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" event={"ID":"07aa5b38-7934-4ff6-baff-0bdf18326138","Type":"ContainerDied","Data":"9c5fab005e67237e5d205d153a50227e55cd722b2e1ae9379cebee9d9542d4e5"} Jul 24 00:18:08.062337 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:08.062303 2576 generic.go:356] "Generic (PLEG): container finished" podID="07aa5b38-7934-4ff6-baff-0bdf18326138" containerID="b524cf2f86ac39387493d6c035f411eb8eb9f9f57c58220252979934bdaef65a" exitCode=0 Jul 24 00:18:08.062725 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:08.062388 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" event={"ID":"07aa5b38-7934-4ff6-baff-0bdf18326138","Type":"ContainerDied","Data":"b524cf2f86ac39387493d6c035f411eb8eb9f9f57c58220252979934bdaef65a"} Jul 24 00:18:09.190981 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:09.190929 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:18:09.309903 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:09.309864 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5scl\" (UniqueName: \"kubernetes.io/projected/07aa5b38-7934-4ff6-baff-0bdf18326138-kube-api-access-r5scl\") pod \"07aa5b38-7934-4ff6-baff-0bdf18326138\" (UID: \"07aa5b38-7934-4ff6-baff-0bdf18326138\") " Jul 24 00:18:09.310100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:09.309938 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07aa5b38-7934-4ff6-baff-0bdf18326138-util\") pod \"07aa5b38-7934-4ff6-baff-0bdf18326138\" (UID: \"07aa5b38-7934-4ff6-baff-0bdf18326138\") " Jul 24 00:18:09.310100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:09.310005 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07aa5b38-7934-4ff6-baff-0bdf18326138-bundle\") pod \"07aa5b38-7934-4ff6-baff-0bdf18326138\" (UID: \"07aa5b38-7934-4ff6-baff-0bdf18326138\") " Jul 24 00:18:09.310606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:09.310581 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07aa5b38-7934-4ff6-baff-0bdf18326138-bundle" (OuterVolumeSpecName: "bundle") pod "07aa5b38-7934-4ff6-baff-0bdf18326138" (UID: "07aa5b38-7934-4ff6-baff-0bdf18326138"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:18:09.312105 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:09.312082 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07aa5b38-7934-4ff6-baff-0bdf18326138-kube-api-access-r5scl" (OuterVolumeSpecName: "kube-api-access-r5scl") pod "07aa5b38-7934-4ff6-baff-0bdf18326138" (UID: "07aa5b38-7934-4ff6-baff-0bdf18326138"). InnerVolumeSpecName "kube-api-access-r5scl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:18:09.314958 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:09.314916 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07aa5b38-7934-4ff6-baff-0bdf18326138-util" (OuterVolumeSpecName: "util") pod "07aa5b38-7934-4ff6-baff-0bdf18326138" (UID: "07aa5b38-7934-4ff6-baff-0bdf18326138"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:18:09.410650 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:09.410567 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5scl\" (UniqueName: \"kubernetes.io/projected/07aa5b38-7934-4ff6-baff-0bdf18326138-kube-api-access-r5scl\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:18:09.410650 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:09.410608 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07aa5b38-7934-4ff6-baff-0bdf18326138-util\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:18:09.410650 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:09.410618 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07aa5b38-7934-4ff6-baff-0bdf18326138-bundle\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:18:10.072179 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:10.072151 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" Jul 24 00:18:10.072359 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:10.072145 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402tkk4" event={"ID":"07aa5b38-7934-4ff6-baff-0bdf18326138","Type":"ContainerDied","Data":"272d7a6ed40e4101fb480141db0a7634989e79832bb52ab48fed2e48ee8c4325"} Jul 24 00:18:10.072359 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:10.072267 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="272d7a6ed40e4101fb480141db0a7634989e79832bb52ab48fed2e48ee8c4325" Jul 24 00:18:17.797736 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.797695 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m"] Jul 24 00:18:17.798330 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.798108 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07aa5b38-7934-4ff6-baff-0bdf18326138" containerName="pull" Jul 24 00:18:17.798330 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.798126 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="07aa5b38-7934-4ff6-baff-0bdf18326138" containerName="pull" Jul 24 00:18:17.798330 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.798140 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07aa5b38-7934-4ff6-baff-0bdf18326138" containerName="extract" Jul 24 00:18:17.798330 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.798148 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="07aa5b38-7934-4ff6-baff-0bdf18326138" containerName="extract" Jul 24 00:18:17.798330 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.798170 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07aa5b38-7934-4ff6-baff-0bdf18326138" containerName="util" Jul 24 00:18:17.798330 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.798177 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="07aa5b38-7934-4ff6-baff-0bdf18326138" containerName="util" Jul 24 00:18:17.798330 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.798252 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="07aa5b38-7934-4ff6-baff-0bdf18326138" containerName="extract" Jul 24 00:18:17.801879 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.801856 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m" Jul 24 00:18:17.804561 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.804539 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 24 00:18:17.805447 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.805431 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9ttm8\"" Jul 24 00:18:17.806204 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.806189 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:18:17.818184 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.818158 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m"] Jul 24 00:18:17.879455 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.879397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mjk2m\" (UID: \"4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m" Jul 24 00:18:17.879617 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.879480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xwwb\" (UniqueName: \"kubernetes.io/projected/4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54-kube-api-access-2xwwb\") pod \"cert-manager-operator-controller-manager-55c444bf86-mjk2m\" (UID: \"4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m" Jul 24 00:18:17.980746 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.980707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mjk2m\" (UID: \"4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m" Jul 24 00:18:17.980746 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.980747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xwwb\" (UniqueName: \"kubernetes.io/projected/4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54-kube-api-access-2xwwb\") pod \"cert-manager-operator-controller-manager-55c444bf86-mjk2m\" (UID: \"4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m" Jul 24 00:18:17.981145 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.981124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mjk2m\" (UID: \"4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m" Jul 24 00:18:17.989231 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:17.989206 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xwwb\" (UniqueName: \"kubernetes.io/projected/4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54-kube-api-access-2xwwb\") pod \"cert-manager-operator-controller-manager-55c444bf86-mjk2m\" (UID: \"4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m" Jul 24 00:18:18.110473 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:18.110378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m" Jul 24 00:18:18.249073 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:18.248938 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m"] Jul 24 00:18:18.255009 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:18:18.254968 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8dcaab_7fe0_44d5_bd6b_76fe9e85ad54.slice/crio-9a97807ee0d878276bf42a60a7cd2ae8cec10c097e1ad3d1f2ecddd688755be2 WatchSource:0}: Error finding container 9a97807ee0d878276bf42a60a7cd2ae8cec10c097e1ad3d1f2ecddd688755be2: Status 404 returned error can't find the container with id 9a97807ee0d878276bf42a60a7cd2ae8cec10c097e1ad3d1f2ecddd688755be2 Jul 24 00:18:19.106260 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:19.106214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m" event={"ID":"4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54","Type":"ContainerStarted","Data":"9a97807ee0d878276bf42a60a7cd2ae8cec10c097e1ad3d1f2ecddd688755be2"} Jul 24 00:18:20.113112 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:20.113027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m" event={"ID":"4a8dcaab-7fe0-44d5-bd6b-76fe9e85ad54","Type":"ContainerStarted","Data":"40483eb6cc37a1a45d97cd47f1940367ebf359baf821b60f8017c51b47c61ff0"} Jul 24 00:18:20.146261 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:20.146209 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mjk2m" podStartSLOduration=1.5437538910000002 podStartE2EDuration="3.146190822s" totalImagesPullingTime="1.602436931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:18:20.14602428 +0000 UTC m=+276.719121082" watchObservedRunningTime="2026-07-24 00:18:20.146190822 +0000 UTC m=+276.719287626" Jul 24 00:18:21.423747 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.423709 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb"] Jul 24 00:18:21.425789 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.425771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:21.429299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.429275 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 00:18:21.429299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.429290 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-j9flg\"" Jul 24 00:18:21.429446 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.429292 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 00:18:21.437053 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.437028 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb"] Jul 24 00:18:21.511212 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.511176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a39567b-04c7-4b42-ac7e-cb509444c47d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb\" (UID: \"2a39567b-04c7-4b42-ac7e-cb509444c47d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:21.511363 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.511226 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bxgt\" (UniqueName: \"kubernetes.io/projected/2a39567b-04c7-4b42-ac7e-cb509444c47d-kube-api-access-7bxgt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb\" (UID: \"2a39567b-04c7-4b42-ac7e-cb509444c47d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:21.511363 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.511285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a39567b-04c7-4b42-ac7e-cb509444c47d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb\" (UID: \"2a39567b-04c7-4b42-ac7e-cb509444c47d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:21.612515 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.612482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a39567b-04c7-4b42-ac7e-cb509444c47d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb\" (UID: \"2a39567b-04c7-4b42-ac7e-cb509444c47d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:21.612705 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.612540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a39567b-04c7-4b42-ac7e-cb509444c47d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb\" (UID: \"2a39567b-04c7-4b42-ac7e-cb509444c47d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:21.612705 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.612574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bxgt\" (UniqueName: \"kubernetes.io/projected/2a39567b-04c7-4b42-ac7e-cb509444c47d-kube-api-access-7bxgt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb\" (UID: \"2a39567b-04c7-4b42-ac7e-cb509444c47d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:21.612941 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.612919 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a39567b-04c7-4b42-ac7e-cb509444c47d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb\" (UID: \"2a39567b-04c7-4b42-ac7e-cb509444c47d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:21.612941 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.612930 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a39567b-04c7-4b42-ac7e-cb509444c47d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb\" (UID: \"2a39567b-04c7-4b42-ac7e-cb509444c47d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:21.622801 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.622780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bxgt\" (UniqueName: \"kubernetes.io/projected/2a39567b-04c7-4b42-ac7e-cb509444c47d-kube-api-access-7bxgt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb\" (UID: \"2a39567b-04c7-4b42-ac7e-cb509444c47d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:21.734983 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.734875 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:21.861610 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:21.861577 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb"] Jul 24 00:18:21.865100 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:18:21.865076 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a39567b_04c7_4b42_ac7e_cb509444c47d.slice/crio-01b7d59f376663c71f9461a0753efc063bba96f9fa0de6cb801737695234bf5d WatchSource:0}: Error finding container 01b7d59f376663c71f9461a0753efc063bba96f9fa0de6cb801737695234bf5d: Status 404 returned error can't find the container with id 01b7d59f376663c71f9461a0753efc063bba96f9fa0de6cb801737695234bf5d Jul 24 00:18:22.124747 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:22.124718 2576 generic.go:356] "Generic (PLEG): container finished" podID="2a39567b-04c7-4b42-ac7e-cb509444c47d" containerID="0c79803efa150f12923746a4a8fc0fb18c8ab6d68e4ea10c925eda3df5c3f0eb" exitCode=0 Jul 24 00:18:22.124906 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:22.124765 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" event={"ID":"2a39567b-04c7-4b42-ac7e-cb509444c47d","Type":"ContainerDied","Data":"0c79803efa150f12923746a4a8fc0fb18c8ab6d68e4ea10c925eda3df5c3f0eb"} Jul 24 00:18:22.124906 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:22.124798 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" event={"ID":"2a39567b-04c7-4b42-ac7e-cb509444c47d","Type":"ContainerStarted","Data":"01b7d59f376663c71f9461a0753efc063bba96f9fa0de6cb801737695234bf5d"} Jul 24 00:18:24.366863 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.366833 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7"] Jul 24 00:18:24.371453 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.371434 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7" Jul 24 00:18:24.375652 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.375629 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 00:18:24.375915 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.375900 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 00:18:24.376228 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.376215 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-md6mx\"" Jul 24 00:18:24.413729 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.413700 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7"] Jul 24 00:18:24.436227 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.436193 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e7df39-cca5-43f3-8f48-fd1be5962925-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gsnh7\" (UID: \"09e7df39-cca5-43f3-8f48-fd1be5962925\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7" Jul 24 00:18:24.436349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.436270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47zc\" (UniqueName: \"kubernetes.io/projected/09e7df39-cca5-43f3-8f48-fd1be5962925-kube-api-access-n47zc\") pod \"cert-manager-cainjector-559cdd8c87-gsnh7\" (UID: \"09e7df39-cca5-43f3-8f48-fd1be5962925\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7" Jul 24 00:18:24.537731 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.537694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e7df39-cca5-43f3-8f48-fd1be5962925-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gsnh7\" (UID: \"09e7df39-cca5-43f3-8f48-fd1be5962925\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7" Jul 24 00:18:24.537906 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.537764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47zc\" (UniqueName: \"kubernetes.io/projected/09e7df39-cca5-43f3-8f48-fd1be5962925-kube-api-access-n47zc\") pod \"cert-manager-cainjector-559cdd8c87-gsnh7\" (UID: \"09e7df39-cca5-43f3-8f48-fd1be5962925\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7" Jul 24 00:18:24.550479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.550446 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e7df39-cca5-43f3-8f48-fd1be5962925-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gsnh7\" (UID: \"09e7df39-cca5-43f3-8f48-fd1be5962925\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7" Jul 24 00:18:24.550938 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.550917 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47zc\" (UniqueName: \"kubernetes.io/projected/09e7df39-cca5-43f3-8f48-fd1be5962925-kube-api-access-n47zc\") pod \"cert-manager-cainjector-559cdd8c87-gsnh7\" (UID: \"09e7df39-cca5-43f3-8f48-fd1be5962925\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7" Jul 24 00:18:24.700995 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.700887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7" Jul 24 00:18:24.830641 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:24.830571 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7"] Jul 24 00:18:24.833027 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:18:24.833000 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09e7df39_cca5_43f3_8f48_fd1be5962925.slice/crio-91d76a90853f5fb1f07c9210fc2dd57ec5e9105a74d4701b9015fc3a4a0149d6 WatchSource:0}: Error finding container 91d76a90853f5fb1f07c9210fc2dd57ec5e9105a74d4701b9015fc3a4a0149d6: Status 404 returned error can't find the container with id 91d76a90853f5fb1f07c9210fc2dd57ec5e9105a74d4701b9015fc3a4a0149d6 Jul 24 00:18:25.138892 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:25.138859 2576 generic.go:356] "Generic (PLEG): container finished" podID="2a39567b-04c7-4b42-ac7e-cb509444c47d" containerID="38d9d8fc5ad207cd425ed36a144439dd89fada6507ed70dae39265d68ceb2d65" exitCode=0 Jul 24 00:18:25.139125 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:25.138966 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" event={"ID":"2a39567b-04c7-4b42-ac7e-cb509444c47d","Type":"ContainerDied","Data":"38d9d8fc5ad207cd425ed36a144439dd89fada6507ed70dae39265d68ceb2d65"} Jul 24 00:18:25.140190 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:25.140168 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7" event={"ID":"09e7df39-cca5-43f3-8f48-fd1be5962925","Type":"ContainerStarted","Data":"91d76a90853f5fb1f07c9210fc2dd57ec5e9105a74d4701b9015fc3a4a0149d6"} Jul 24 00:18:26.147430 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:26.147393 2576 generic.go:356] "Generic (PLEG): container finished" podID="2a39567b-04c7-4b42-ac7e-cb509444c47d" containerID="79821ff6e27abd220d0385d7116f8f0a56646ddb712f89f9d387e4bc30f9b2d6" exitCode=0 Jul 24 00:18:26.147873 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:26.147436 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" event={"ID":"2a39567b-04c7-4b42-ac7e-cb509444c47d","Type":"ContainerDied","Data":"79821ff6e27abd220d0385d7116f8f0a56646ddb712f89f9d387e4bc30f9b2d6"} Jul 24 00:18:27.293758 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:27.293737 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:27.361654 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:27.361622 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bxgt\" (UniqueName: \"kubernetes.io/projected/2a39567b-04c7-4b42-ac7e-cb509444c47d-kube-api-access-7bxgt\") pod \"2a39567b-04c7-4b42-ac7e-cb509444c47d\" (UID: \"2a39567b-04c7-4b42-ac7e-cb509444c47d\") " Jul 24 00:18:27.361789 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:27.361684 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a39567b-04c7-4b42-ac7e-cb509444c47d-util\") pod \"2a39567b-04c7-4b42-ac7e-cb509444c47d\" (UID: \"2a39567b-04c7-4b42-ac7e-cb509444c47d\") " Jul 24 00:18:27.361789 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:27.361754 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a39567b-04c7-4b42-ac7e-cb509444c47d-bundle\") pod \"2a39567b-04c7-4b42-ac7e-cb509444c47d\" (UID: \"2a39567b-04c7-4b42-ac7e-cb509444c47d\") " Jul 24 00:18:27.362144 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:27.362118 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a39567b-04c7-4b42-ac7e-cb509444c47d-bundle" (OuterVolumeSpecName: "bundle") pod "2a39567b-04c7-4b42-ac7e-cb509444c47d" (UID: "2a39567b-04c7-4b42-ac7e-cb509444c47d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:18:27.363733 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:27.363705 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a39567b-04c7-4b42-ac7e-cb509444c47d-kube-api-access-7bxgt" (OuterVolumeSpecName: "kube-api-access-7bxgt") pod "2a39567b-04c7-4b42-ac7e-cb509444c47d" (UID: "2a39567b-04c7-4b42-ac7e-cb509444c47d"). InnerVolumeSpecName "kube-api-access-7bxgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:18:27.367018 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:27.366995 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a39567b-04c7-4b42-ac7e-cb509444c47d-util" (OuterVolumeSpecName: "util") pod "2a39567b-04c7-4b42-ac7e-cb509444c47d" (UID: "2a39567b-04c7-4b42-ac7e-cb509444c47d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:18:27.462768 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:27.462676 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a39567b-04c7-4b42-ac7e-cb509444c47d-util\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:18:27.462768 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:27.462718 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a39567b-04c7-4b42-ac7e-cb509444c47d-bundle\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:18:27.462768 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:27.462734 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bxgt\" (UniqueName: \"kubernetes.io/projected/2a39567b-04c7-4b42-ac7e-cb509444c47d-kube-api-access-7bxgt\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:18:28.157928 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:28.157892 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7" event={"ID":"09e7df39-cca5-43f3-8f48-fd1be5962925","Type":"ContainerStarted","Data":"b1ad365281cb070b0bde0ffb24ac8d5225843f863604a3fb59908a6a2cd5de83"} Jul 24 00:18:28.160147 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:28.160118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" event={"ID":"2a39567b-04c7-4b42-ac7e-cb509444c47d","Type":"ContainerDied","Data":"01b7d59f376663c71f9461a0753efc063bba96f9fa0de6cb801737695234bf5d"} Jul 24 00:18:28.160271 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:28.160157 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmmxnb" Jul 24 00:18:28.160271 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:28.160166 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01b7d59f376663c71f9461a0753efc063bba96f9fa0de6cb801737695234bf5d" Jul 24 00:18:28.173311 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:28.173263 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gsnh7" podStartSLOduration=1.7281679730000001 podStartE2EDuration="4.173246118s" totalImagesPullingTime="2.445078145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:18:28.172057197 +0000 UTC m=+284.745154013" watchObservedRunningTime="2026-07-24 00:18:28.173246118 +0000 UTC m=+284.746342921" Jul 24 00:18:32.003438 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.003406 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8"] Jul 24 00:18:32.003803 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.003676 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a39567b-04c7-4b42-ac7e-cb509444c47d" containerName="util" Jul 24 00:18:32.003803 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.003686 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a39567b-04c7-4b42-ac7e-cb509444c47d" containerName="util" Jul 24 00:18:32.003803 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.003702 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a39567b-04c7-4b42-ac7e-cb509444c47d" containerName="pull" Jul 24 00:18:32.003803 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.003709 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a39567b-04c7-4b42-ac7e-cb509444c47d" containerName="pull" Jul 24 00:18:32.003803 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.003721 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a39567b-04c7-4b42-ac7e-cb509444c47d" containerName="extract" Jul 24 00:18:32.003803 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.003726 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a39567b-04c7-4b42-ac7e-cb509444c47d" containerName="extract" Jul 24 00:18:32.003803 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.003770 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a39567b-04c7-4b42-ac7e-cb509444c47d" containerName="extract" Jul 24 00:18:32.007019 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.007002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8" Jul 24 00:18:32.010984 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.010962 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 00:18:32.011661 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.011642 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-p666g\"" Jul 24 00:18:32.011661 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.011656 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:18:32.037009 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.036977 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8"] Jul 24 00:18:32.097800 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.097770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcqs9\" (UniqueName: \"kubernetes.io/projected/a5718c65-142e-4729-b61d-50e9fca7d888-kube-api-access-fcqs9\") pod \"openshift-lws-operator-fd8ccff4c-4vml8\" (UID: \"a5718c65-142e-4729-b61d-50e9fca7d888\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8" Jul 24 00:18:32.097997 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.097910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5718c65-142e-4729-b61d-50e9fca7d888-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4vml8\" (UID: \"a5718c65-142e-4729-b61d-50e9fca7d888\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8" Jul 24 00:18:32.198601 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.198560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqs9\" (UniqueName: \"kubernetes.io/projected/a5718c65-142e-4729-b61d-50e9fca7d888-kube-api-access-fcqs9\") pod \"openshift-lws-operator-fd8ccff4c-4vml8\" (UID: \"a5718c65-142e-4729-b61d-50e9fca7d888\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8" Jul 24 00:18:32.198765 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.198660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5718c65-142e-4729-b61d-50e9fca7d888-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4vml8\" (UID: \"a5718c65-142e-4729-b61d-50e9fca7d888\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8" Jul 24 00:18:32.199159 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.199132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5718c65-142e-4729-b61d-50e9fca7d888-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4vml8\" (UID: \"a5718c65-142e-4729-b61d-50e9fca7d888\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8" Jul 24 00:18:32.211120 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.211085 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcqs9\" (UniqueName: \"kubernetes.io/projected/a5718c65-142e-4729-b61d-50e9fca7d888-kube-api-access-fcqs9\") pod \"openshift-lws-operator-fd8ccff4c-4vml8\" (UID: \"a5718c65-142e-4729-b61d-50e9fca7d888\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8" Jul 24 00:18:32.315584 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.315553 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8" Jul 24 00:18:32.445902 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:32.445878 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8"] Jul 24 00:18:32.448070 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:18:32.448041 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5718c65_142e_4729_b61d_50e9fca7d888.slice/crio-47f594c3d47b15a71544694862c1b293e9b9e0cda2fa925f77545c40ae703308 WatchSource:0}: Error finding container 47f594c3d47b15a71544694862c1b293e9b9e0cda2fa925f77545c40ae703308: Status 404 returned error can't find the container with id 47f594c3d47b15a71544694862c1b293e9b9e0cda2fa925f77545c40ae703308 Jul 24 00:18:33.181316 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:33.181275 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8" event={"ID":"a5718c65-142e-4729-b61d-50e9fca7d888","Type":"ContainerStarted","Data":"47f594c3d47b15a71544694862c1b293e9b9e0cda2fa925f77545c40ae703308"} Jul 24 00:18:35.191763 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:35.191725 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8" event={"ID":"a5718c65-142e-4729-b61d-50e9fca7d888","Type":"ContainerStarted","Data":"8c43ec79bb3b12e8077e7464aa753e3b4c9529013e37ed639775b311bcbed7b2"} Jul 24 00:18:35.207738 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:35.207687 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4vml8" podStartSLOduration=2.310945308 podStartE2EDuration="4.207673678s" totalImagesPullingTime="1.89672837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:18:35.206277421 +0000 UTC m=+291.779374225" watchObservedRunningTime="2026-07-24 00:18:35.207673678 +0000 UTC m=+291.780770484" Jul 24 00:18:38.083805 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.083762 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd"] Jul 24 00:18:38.108929 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.108891 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd"] Jul 24 00:18:38.109100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.108969 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:38.111645 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.111624 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 00:18:38.111645 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.111633 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-j9flg\"" Jul 24 00:18:38.112333 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.112317 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 00:18:38.252362 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.252319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd\" (UID: \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:38.252362 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.252362 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd\" (UID: \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:38.252591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.252417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zb9\" (UniqueName: \"kubernetes.io/projected/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-kube-api-access-46zb9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd\" (UID: \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:38.353802 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.353721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zb9\" (UniqueName: \"kubernetes.io/projected/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-kube-api-access-46zb9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd\" (UID: \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:38.353802 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.353775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd\" (UID: \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:38.353802 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.353794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd\" (UID: \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:38.354146 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.354130 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd\" (UID: \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:38.354183 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.354165 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd\" (UID: \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:38.361528 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.361500 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zb9\" (UniqueName: \"kubernetes.io/projected/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-kube-api-access-46zb9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd\" (UID: \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:38.417824 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.417787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:38.553779 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:38.553749 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd"] Jul 24 00:18:38.556315 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:18:38.556286 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716b8f0d_cd4d_47cb_aa81_2c64fb0bb849.slice/crio-0b86aaffb677709a484fb72e355946757f1b80cba4dac90bec6fbf16fa2f8d12 WatchSource:0}: Error finding container 0b86aaffb677709a484fb72e355946757f1b80cba4dac90bec6fbf16fa2f8d12: Status 404 returned error can't find the container with id 0b86aaffb677709a484fb72e355946757f1b80cba4dac90bec6fbf16fa2f8d12 Jul 24 00:18:39.210252 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:39.210215 2576 generic.go:356] "Generic (PLEG): container finished" podID="716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" containerID="57d0f871b0a8e21d1c90c0e923d01abeae1ccc77abfdf5b2209bf95e6951a53f" exitCode=0 Jul 24 00:18:39.210661 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:39.210306 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" event={"ID":"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849","Type":"ContainerDied","Data":"57d0f871b0a8e21d1c90c0e923d01abeae1ccc77abfdf5b2209bf95e6951a53f"} Jul 24 00:18:39.210661 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:39.210368 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" event={"ID":"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849","Type":"ContainerStarted","Data":"0b86aaffb677709a484fb72e355946757f1b80cba4dac90bec6fbf16fa2f8d12"} Jul 24 00:18:40.216832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:40.216743 2576 generic.go:356] "Generic (PLEG): container finished" podID="716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" containerID="9bc88f8437f0ab5b1f9153279fb3ce422e9f8ce0eaee4db2b08854f314582dad" exitCode=0 Jul 24 00:18:40.217248 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:40.216830 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" event={"ID":"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849","Type":"ContainerDied","Data":"9bc88f8437f0ab5b1f9153279fb3ce422e9f8ce0eaee4db2b08854f314582dad"} Jul 24 00:18:41.223374 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:41.223335 2576 generic.go:356] "Generic (PLEG): container finished" podID="716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" containerID="35905bac9e6b1369fe67af02c8b24398305dfc1077a2183622a4502b7db10d94" exitCode=0 Jul 24 00:18:41.223784 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:41.223421 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" event={"ID":"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849","Type":"ContainerDied","Data":"35905bac9e6b1369fe67af02c8b24398305dfc1077a2183622a4502b7db10d94"} Jul 24 00:18:42.354414 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:42.354389 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:42.486399 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:42.486300 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-bundle\") pod \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\" (UID: \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\") " Jul 24 00:18:42.486399 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:42.486383 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46zb9\" (UniqueName: \"kubernetes.io/projected/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-kube-api-access-46zb9\") pod \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\" (UID: \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\") " Jul 24 00:18:42.486642 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:42.486418 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-util\") pod \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\" (UID: \"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849\") " Jul 24 00:18:42.487112 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:42.487084 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-bundle" (OuterVolumeSpecName: "bundle") pod "716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" (UID: "716b8f0d-cd4d-47cb-aa81-2c64fb0bb849"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:18:42.488551 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:42.488529 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-kube-api-access-46zb9" (OuterVolumeSpecName: "kube-api-access-46zb9") pod "716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" (UID: "716b8f0d-cd4d-47cb-aa81-2c64fb0bb849"). InnerVolumeSpecName "kube-api-access-46zb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:18:42.491974 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:42.491939 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-util" (OuterVolumeSpecName: "util") pod "716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" (UID: "716b8f0d-cd4d-47cb-aa81-2c64fb0bb849"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:18:42.587158 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:42.587110 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-bundle\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:18:42.587158 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:42.587151 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46zb9\" (UniqueName: \"kubernetes.io/projected/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-kube-api-access-46zb9\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:18:42.587158 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:42.587160 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/716b8f0d-cd4d-47cb-aa81-2c64fb0bb849-util\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:18:43.234495 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:43.234457 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" Jul 24 00:18:43.234495 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:43.234469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqdfpd" event={"ID":"716b8f0d-cd4d-47cb-aa81-2c64fb0bb849","Type":"ContainerDied","Data":"0b86aaffb677709a484fb72e355946757f1b80cba4dac90bec6fbf16fa2f8d12"} Jul 24 00:18:43.234773 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:43.234513 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b86aaffb677709a484fb72e355946757f1b80cba4dac90bec6fbf16fa2f8d12" Jul 24 00:18:43.864531 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:43.864503 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:18:43.867230 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:43.867206 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:18:43.875076 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:43.875048 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 00:18:54.782344 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.782304 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks"] Jul 24 00:18:54.786925 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.782610 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" containerName="util" Jul 24 00:18:54.786925 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.782622 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" containerName="util" Jul 24 00:18:54.786925 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.782637 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" containerName="pull" Jul 24 00:18:54.786925 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.782643 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" containerName="pull" Jul 24 00:18:54.786925 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.782653 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" containerName="extract" Jul 24 00:18:54.786925 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.782659 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" containerName="extract" Jul 24 00:18:54.786925 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.782704 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="716b8f0d-cd4d-47cb-aa81-2c64fb0bb849" containerName="extract" Jul 24 00:18:54.787565 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.787546 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.789884 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.789861 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 00:18:54.789884 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.789877 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 00:18:54.790541 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.790519 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 00:18:54.790668 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.790601 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 00:18:54.790747 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.790677 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b26lx\"" Jul 24 00:18:54.799582 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.799553 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks"] Jul 24 00:18:54.890623 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.890579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e127dce8-7b34-4e2e-a8d0-861ca1627220-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.890623 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.890624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e127dce8-7b34-4e2e-a8d0-861ca1627220-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.890838 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.890694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e127dce8-7b34-4e2e-a8d0-861ca1627220-tmp\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.890838 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.890727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbkl9\" (UniqueName: \"kubernetes.io/projected/e127dce8-7b34-4e2e-a8d0-861ca1627220-kube-api-access-mbkl9\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.890838 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.890765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e127dce8-7b34-4e2e-a8d0-861ca1627220-manifests\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.991962 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.991922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e127dce8-7b34-4e2e-a8d0-861ca1627220-tmp\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.992124 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.992001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkl9\" (UniqueName: \"kubernetes.io/projected/e127dce8-7b34-4e2e-a8d0-861ca1627220-kube-api-access-mbkl9\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.992124 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.992042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e127dce8-7b34-4e2e-a8d0-861ca1627220-manifests\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.992124 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.992076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e127dce8-7b34-4e2e-a8d0-861ca1627220-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.992124 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.992105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e127dce8-7b34-4e2e-a8d0-861ca1627220-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.992461 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.992438 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e127dce8-7b34-4e2e-a8d0-861ca1627220-manifests\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.994246 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.994210 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e127dce8-7b34-4e2e-a8d0-861ca1627220-tmp\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.994459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.994439 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e127dce8-7b34-4e2e-a8d0-861ca1627220-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:54.994520 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:54.994446 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e127dce8-7b34-4e2e-a8d0-861ca1627220-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:55.001383 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:55.001352 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbkl9\" (UniqueName: \"kubernetes.io/projected/e127dce8-7b34-4e2e-a8d0-861ca1627220-kube-api-access-mbkl9\") pod \"opendatahub-operator-controller-manager-6cff767bd6-8zhks\" (UID: \"e127dce8-7b34-4e2e-a8d0-861ca1627220\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:55.098257 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:55.098217 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:18:55.247849 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:55.247728 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks"] Jul 24 00:18:55.250299 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:18:55.250269 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode127dce8_7b34_4e2e_a8d0_861ca1627220.slice/crio-9cb84c939877a4791682c9704f5be7c94a727b2cc36ddbde4faa772478e92366 WatchSource:0}: Error finding container 9cb84c939877a4791682c9704f5be7c94a727b2cc36ddbde4faa772478e92366: Status 404 returned error can't find the container with id 9cb84c939877a4791682c9704f5be7c94a727b2cc36ddbde4faa772478e92366 Jul 24 00:18:55.252253 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:55.252234 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 00:18:55.286350 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:55.286315 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" event={"ID":"e127dce8-7b34-4e2e-a8d0-861ca1627220","Type":"ContainerStarted","Data":"9cb84c939877a4791682c9704f5be7c94a727b2cc36ddbde4faa772478e92366"} Jul 24 00:18:58.306027 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:58.305987 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" event={"ID":"e127dce8-7b34-4e2e-a8d0-861ca1627220","Type":"ContainerStarted","Data":"992c68b15d7673e335097e335e77c6bbe44833a84b26dddb4d3e8e595f7a05bd"} Jul 24 00:18:59.311925 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:59.311890 2576 generic.go:356] "Generic (PLEG): container finished" podID="e127dce8-7b34-4e2e-a8d0-861ca1627220" containerID="992c68b15d7673e335097e335e77c6bbe44833a84b26dddb4d3e8e595f7a05bd" exitCode=0 Jul 24 00:18:59.312323 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:18:59.311981 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" event={"ID":"e127dce8-7b34-4e2e-a8d0-861ca1627220","Type":"ContainerDied","Data":"992c68b15d7673e335097e335e77c6bbe44833a84b26dddb4d3e8e595f7a05bd"} Jul 24 00:19:00.318395 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:00.318358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" event={"ID":"e127dce8-7b34-4e2e-a8d0-861ca1627220","Type":"ContainerStarted","Data":"f8d1eed9b01d037e905ec65e1e827af2fab8c1fa72d8c073d0640b85adde7d8e"} Jul 24 00:19:00.318812 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:00.318499 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:19:00.343203 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:00.343149 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" podStartSLOduration=2.106318741 podStartE2EDuration="6.343130868s" totalImagesPullingTime="3.227318091s" totalInitContainerRuntime="1.009494036s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:00.340685617 +0000 UTC m=+316.913782419" watchObservedRunningTime="2026-07-24 00:19:00.343130868 +0000 UTC m=+316.916227649" Jul 24 00:19:11.324657 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:11.324627 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-8zhks" Jul 24 00:19:20.129175 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.129135 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp"] Jul 24 00:19:20.133575 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.133557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" Jul 24 00:19:20.148629 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.148597 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 00:19:20.148746 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.148602 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 00:19:20.150118 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.150104 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fms4b\"" Jul 24 00:19:20.155471 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.155448 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp"] Jul 24 00:19:20.162546 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.162517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa64dff6-1958-4442-a3ab-71c1fbe98f86-tmp\") pod \"kube-auth-proxy-6c994f5cd9-9d7vp\" (UID: \"fa64dff6-1958-4442-a3ab-71c1fbe98f86\") " pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" Jul 24 00:19:20.162652 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.162556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9k2t\" (UniqueName: \"kubernetes.io/projected/fa64dff6-1958-4442-a3ab-71c1fbe98f86-kube-api-access-q9k2t\") pod \"kube-auth-proxy-6c994f5cd9-9d7vp\" (UID: \"fa64dff6-1958-4442-a3ab-71c1fbe98f86\") " pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" Jul 24 00:19:20.162652 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.162629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa64dff6-1958-4442-a3ab-71c1fbe98f86-tls-certs\") pod \"kube-auth-proxy-6c994f5cd9-9d7vp\" (UID: \"fa64dff6-1958-4442-a3ab-71c1fbe98f86\") " pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" Jul 24 00:19:20.263868 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.263825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa64dff6-1958-4442-a3ab-71c1fbe98f86-tmp\") pod \"kube-auth-proxy-6c994f5cd9-9d7vp\" (UID: \"fa64dff6-1958-4442-a3ab-71c1fbe98f86\") " pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" Jul 24 00:19:20.263868 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.263867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9k2t\" (UniqueName: \"kubernetes.io/projected/fa64dff6-1958-4442-a3ab-71c1fbe98f86-kube-api-access-q9k2t\") pod \"kube-auth-proxy-6c994f5cd9-9d7vp\" (UID: \"fa64dff6-1958-4442-a3ab-71c1fbe98f86\") " pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" Jul 24 00:19:20.264168 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.263892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa64dff6-1958-4442-a3ab-71c1fbe98f86-tls-certs\") pod \"kube-auth-proxy-6c994f5cd9-9d7vp\" (UID: \"fa64dff6-1958-4442-a3ab-71c1fbe98f86\") " pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" Jul 24 00:19:20.266257 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.266236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa64dff6-1958-4442-a3ab-71c1fbe98f86-tmp\") pod \"kube-auth-proxy-6c994f5cd9-9d7vp\" (UID: \"fa64dff6-1958-4442-a3ab-71c1fbe98f86\") " pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" Jul 24 00:19:20.266490 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.266469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa64dff6-1958-4442-a3ab-71c1fbe98f86-tls-certs\") pod \"kube-auth-proxy-6c994f5cd9-9d7vp\" (UID: \"fa64dff6-1958-4442-a3ab-71c1fbe98f86\") " pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" Jul 24 00:19:20.291887 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.291855 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9k2t\" (UniqueName: \"kubernetes.io/projected/fa64dff6-1958-4442-a3ab-71c1fbe98f86-kube-api-access-q9k2t\") pod \"kube-auth-proxy-6c994f5cd9-9d7vp\" (UID: \"fa64dff6-1958-4442-a3ab-71c1fbe98f86\") " pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" Jul 24 00:19:20.443467 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.443376 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" Jul 24 00:19:20.596234 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:20.596204 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp"] Jul 24 00:19:20.598055 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:19:20.598020 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa64dff6_1958_4442_a3ab_71c1fbe98f86.slice/crio-5b9792a50f0b8a8722d5844b09ecb224a227270616643cb6294066aa5f961674 WatchSource:0}: Error finding container 5b9792a50f0b8a8722d5844b09ecb224a227270616643cb6294066aa5f961674: Status 404 returned error can't find the container with id 5b9792a50f0b8a8722d5844b09ecb224a227270616643cb6294066aa5f961674 Jul 24 00:19:21.420555 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.420509 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" event={"ID":"fa64dff6-1958-4442-a3ab-71c1fbe98f86","Type":"ContainerStarted","Data":"5b9792a50f0b8a8722d5844b09ecb224a227270616643cb6294066aa5f961674"} Jul 24 00:19:21.879322 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.879288 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5"] Jul 24 00:19:21.882923 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.882896 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:21.885232 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.885209 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 00:19:21.885446 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.885373 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 00:19:21.885605 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.885373 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 00:19:21.885705 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.885612 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 00:19:21.885705 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.885626 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-k8vmm\"" Jul 24 00:19:21.895792 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.895772 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5"] Jul 24 00:19:21.974264 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.974232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:21.974264 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.974266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:21.974458 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.974295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:21.974458 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.974330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnkwn\" (UniqueName: \"kubernetes.io/projected/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-kube-api-access-fnkwn\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:21.974458 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.974364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:21.974458 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.974393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:21.974458 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:21.974422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.075101 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.075065 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.075386 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.075119 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.075386 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.075163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnkwn\" (UniqueName: \"kubernetes.io/projected/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-kube-api-access-fnkwn\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.075386 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.075370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.075653 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.075420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.075653 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.075461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.075653 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.075527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.076343 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.076308 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.078083 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.078060 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.078377 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.078353 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.078490 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.078399 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.078551 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.078510 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.086898 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.086854 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnkwn\" (UniqueName: \"kubernetes.io/projected/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-kube-api-access-fnkwn\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.087182 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.087166 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5199901e-f6f0-4c4e-bd1e-dc1a1835766b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-n29p5\" (UID: \"5199901e-f6f0-4c4e-bd1e-dc1a1835766b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.195154 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.195060 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:22.357542 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.357507 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5"] Jul 24 00:19:22.361396 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:19:22.361360 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5199901e_f6f0_4c4e_bd1e_dc1a1835766b.slice/crio-f1c78747f7a4c99c8d1f21eb1f598027e0e428639de0ea861aa831403480b1ac WatchSource:0}: Error finding container f1c78747f7a4c99c8d1f21eb1f598027e0e428639de0ea861aa831403480b1ac: Status 404 returned error can't find the container with id f1c78747f7a4c99c8d1f21eb1f598027e0e428639de0ea861aa831403480b1ac Jul 24 00:19:22.426646 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:22.426614 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" event={"ID":"5199901e-f6f0-4c4e-bd1e-dc1a1835766b","Type":"ContainerStarted","Data":"f1c78747f7a4c99c8d1f21eb1f598027e0e428639de0ea861aa831403480b1ac"} Jul 24 00:19:24.437759 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:24.437716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" event={"ID":"fa64dff6-1958-4442-a3ab-71c1fbe98f86","Type":"ContainerStarted","Data":"58204a24dce61c8307112fe2b4feb228f945f6e50100ce839566859cc0318ca9"} Jul 24 00:19:24.474421 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:24.474358 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c994f5cd9-9d7vp" podStartSLOduration=1.52113934 podStartE2EDuration="4.474342881s" totalImagesPullingTime="2.953203541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:24.471928788 +0000 UTC m=+341.045025591" watchObservedRunningTime="2026-07-24 00:19:24.474342881 +0000 UTC m=+341.047439683" Jul 24 00:19:29.739653 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:29.739617 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:19:29.739894 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:29.739689 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:19:30.472386 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.472341 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" event={"ID":"5199901e-f6f0-4c4e-bd1e-dc1a1835766b","Type":"ContainerStarted","Data":"8284e617917354febe0bd1a8fb383620d5d8e7e0cb2773d70c7273db0500bdd1"} Jul 24 00:19:30.472633 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.472608 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:30.474444 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.474414 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-n29p5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 00:19:30.474575 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.474549 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" podUID="5199901e-f6f0-4c4e-bd1e-dc1a1835766b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 00:19:30.508450 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.508392 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" podStartSLOduration=2.132815588 podStartE2EDuration="9.50837614s" totalImagesPullingTime="7.375560552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:30.506118019 +0000 UTC m=+347.079214822" watchObservedRunningTime="2026-07-24 00:19:30.50837614 +0000 UTC m=+347.081472941" Jul 24 00:19:30.761539 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.761456 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf"] Jul 24 00:19:30.765098 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.765076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.768246 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.768228 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-9btz8\"" Jul 24 00:19:30.817092 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.817057 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf"] Jul 24 00:19:30.864813 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.864778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.864813 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.864813 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc4049d-653f-4735-bd2d-75581abf9202-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.865074 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.864906 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.865074 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.864941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.865074 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.864996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.865074 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.865037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc4049d-653f-4735-bd2d-75581abf9202-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.865241 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.865103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9txq\" (UniqueName: \"kubernetes.io/projected/3cc4049d-653f-4735-bd2d-75581abf9202-kube-api-access-z9txq\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.865241 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.865131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc4049d-653f-4735-bd2d-75581abf9202-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.865241 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.865172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966019 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.965984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966019 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.966024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc4049d-653f-4735-bd2d-75581abf9202-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966262 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.966092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966262 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.966118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966262 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.966146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966262 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.966171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc4049d-653f-4735-bd2d-75581abf9202-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966262 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.966205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9txq\" (UniqueName: \"kubernetes.io/projected/3cc4049d-653f-4735-bd2d-75581abf9202-kube-api-access-z9txq\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966262 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.966235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc4049d-653f-4735-bd2d-75581abf9202-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966526 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.966397 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966526 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.966470 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966526 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.966496 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.966718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.966897 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.966876 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc4049d-653f-4735-bd2d-75581abf9202-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.967026 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.967012 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.968393 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.968368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc4049d-653f-4735-bd2d-75581abf9202-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.968494 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.968473 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc4049d-653f-4735-bd2d-75581abf9202-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.975934 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.975902 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc4049d-653f-4735-bd2d-75581abf9202-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:30.976376 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:30.976356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9txq\" (UniqueName: \"kubernetes.io/projected/3cc4049d-653f-4735-bd2d-75581abf9202-kube-api-access-z9txq\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf\" (UID: \"3cc4049d-653f-4735-bd2d-75581abf9202\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:31.074554 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:31.074506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:31.218594 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:31.218569 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf"] Jul 24 00:19:31.220611 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:19:31.220583 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc4049d_653f_4735_bd2d_75581abf9202.slice/crio-63db53d6c5d96b02ae6762a13db2487c76b6443db5376c89116cb3c6f44fa51a WatchSource:0}: Error finding container 63db53d6c5d96b02ae6762a13db2487c76b6443db5376c89116cb3c6f44fa51a: Status 404 returned error can't find the container with id 63db53d6c5d96b02ae6762a13db2487c76b6443db5376c89116cb3c6f44fa51a Jul 24 00:19:31.477676 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:31.477579 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" event={"ID":"3cc4049d-653f-4735-bd2d-75581abf9202","Type":"ContainerStarted","Data":"63db53d6c5d96b02ae6762a13db2487c76b6443db5376c89116cb3c6f44fa51a"} Jul 24 00:19:31.478541 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:31.478519 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-n29p5" Jul 24 00:19:33.889908 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:33.889854 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:19:33.890191 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:33.889992 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:19:33.890191 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:33.890025 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:19:34.496031 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:34.495987 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" event={"ID":"3cc4049d-653f-4735-bd2d-75581abf9202","Type":"ContainerStarted","Data":"5460874291adce454fd7c2f21fa386dd77f4c0d79ef52bfa8a74c120a22bbaec"} Jul 24 00:19:34.515921 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:34.515870 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" podStartSLOduration=1.848912503 podStartE2EDuration="4.515854765s" totalImagesPullingTime="2.666942262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:34.514321148 +0000 UTC m=+351.087417953" watchObservedRunningTime="2026-07-24 00:19:34.515854765 +0000 UTC m=+351.088951569" Jul 24 00:19:35.074877 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:35.074839 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:35.079719 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:35.079689 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:35.502530 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:35.502431 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:19:35.503507 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:19:35.503486 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf" Jul 24 00:20:15.750084 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:15.750037 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-59brs"] Jul 24 00:20:15.753053 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:15.753036 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-59brs" Jul 24 00:20:15.756091 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:15.756068 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 00:20:15.757089 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:15.757067 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-drhfd\"" Jul 24 00:20:15.757185 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:15.757066 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 00:20:15.761423 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:15.761403 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-59brs"] Jul 24 00:20:15.845108 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:15.845059 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgkcj\" (UniqueName: \"kubernetes.io/projected/8eecb5ef-1eb9-4146-8ef3-ec885ab259d5-kube-api-access-rgkcj\") pod \"kuadrant-operator-catalog-59brs\" (UID: \"8eecb5ef-1eb9-4146-8ef3-ec885ab259d5\") " pod="kuadrant-system/kuadrant-operator-catalog-59brs" Jul 24 00:20:15.946365 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:15.946326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgkcj\" (UniqueName: \"kubernetes.io/projected/8eecb5ef-1eb9-4146-8ef3-ec885ab259d5-kube-api-access-rgkcj\") pod \"kuadrant-operator-catalog-59brs\" (UID: \"8eecb5ef-1eb9-4146-8ef3-ec885ab259d5\") " pod="kuadrant-system/kuadrant-operator-catalog-59brs" Jul 24 00:20:15.960172 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:15.960139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgkcj\" (UniqueName: \"kubernetes.io/projected/8eecb5ef-1eb9-4146-8ef3-ec885ab259d5-kube-api-access-rgkcj\") pod \"kuadrant-operator-catalog-59brs\" (UID: \"8eecb5ef-1eb9-4146-8ef3-ec885ab259d5\") " pod="kuadrant-system/kuadrant-operator-catalog-59brs" Jul 24 00:20:16.062789 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:16.062753 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-59brs" Jul 24 00:20:16.101731 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:16.101693 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-59brs"] Jul 24 00:20:16.199233 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:16.199205 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-59brs"] Jul 24 00:20:16.201483 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:20:16.201453 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eecb5ef_1eb9_4146_8ef3_ec885ab259d5.slice/crio-adfc8f52dc1edde5a504451ca8db03ecf3cb0fbb6a84d95e68f1538a62b8ee88 WatchSource:0}: Error finding container adfc8f52dc1edde5a504451ca8db03ecf3cb0fbb6a84d95e68f1538a62b8ee88: Status 404 returned error can't find the container with id adfc8f52dc1edde5a504451ca8db03ecf3cb0fbb6a84d95e68f1538a62b8ee88 Jul 24 00:20:16.312553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:16.312516 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vx7gf"] Jul 24 00:20:16.316643 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:16.316598 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vx7gf" Jul 24 00:20:16.324152 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:16.324124 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vx7gf"] Jul 24 00:20:16.350463 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:16.350436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw9gj\" (UniqueName: \"kubernetes.io/projected/2eb870b4-8978-43c6-bcb5-f4b96253989d-kube-api-access-xw9gj\") pod \"kuadrant-operator-catalog-vx7gf\" (UID: \"2eb870b4-8978-43c6-bcb5-f4b96253989d\") " pod="kuadrant-system/kuadrant-operator-catalog-vx7gf" Jul 24 00:20:16.450939 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:16.450908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw9gj\" (UniqueName: \"kubernetes.io/projected/2eb870b4-8978-43c6-bcb5-f4b96253989d-kube-api-access-xw9gj\") pod \"kuadrant-operator-catalog-vx7gf\" (UID: \"2eb870b4-8978-43c6-bcb5-f4b96253989d\") " pod="kuadrant-system/kuadrant-operator-catalog-vx7gf" Jul 24 00:20:16.459534 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:16.459506 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw9gj\" (UniqueName: \"kubernetes.io/projected/2eb870b4-8978-43c6-bcb5-f4b96253989d-kube-api-access-xw9gj\") pod \"kuadrant-operator-catalog-vx7gf\" (UID: \"2eb870b4-8978-43c6-bcb5-f4b96253989d\") " pod="kuadrant-system/kuadrant-operator-catalog-vx7gf" Jul 24 00:20:16.626471 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:16.626386 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vx7gf" Jul 24 00:20:16.686774 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:16.686735 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-59brs" event={"ID":"8eecb5ef-1eb9-4146-8ef3-ec885ab259d5","Type":"ContainerStarted","Data":"adfc8f52dc1edde5a504451ca8db03ecf3cb0fbb6a84d95e68f1538a62b8ee88"} Jul 24 00:20:16.759637 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:16.759600 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vx7gf"] Jul 24 00:20:16.774858 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:20:16.774823 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb870b4_8978_43c6_bcb5_f4b96253989d.slice/crio-8df63296cbd2a52694ba870c8b3c2a829cc55ccf5888665b8500840b4b89c152 WatchSource:0}: Error finding container 8df63296cbd2a52694ba870c8b3c2a829cc55ccf5888665b8500840b4b89c152: Status 404 returned error can't find the container with id 8df63296cbd2a52694ba870c8b3c2a829cc55ccf5888665b8500840b4b89c152 Jul 24 00:20:17.695968 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:17.695915 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vx7gf" event={"ID":"2eb870b4-8978-43c6-bcb5-f4b96253989d","Type":"ContainerStarted","Data":"8df63296cbd2a52694ba870c8b3c2a829cc55ccf5888665b8500840b4b89c152"} Jul 24 00:20:18.702450 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:18.702323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-59brs" event={"ID":"8eecb5ef-1eb9-4146-8ef3-ec885ab259d5","Type":"ContainerStarted","Data":"1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc"} Jul 24 00:20:18.702450 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:18.702367 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-59brs" podUID="8eecb5ef-1eb9-4146-8ef3-ec885ab259d5" containerName="registry-server" containerID="cri-o://1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc" gracePeriod=2 Jul 24 00:20:18.703810 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:18.703784 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vx7gf" event={"ID":"2eb870b4-8978-43c6-bcb5-f4b96253989d","Type":"ContainerStarted","Data":"d0545f90f9b1fd37ceb3e00fce891eb16d143474e73537f1d23a51d3d1655e63"} Jul 24 00:20:18.718066 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:18.718025 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-59brs" podStartSLOduration=1.470891843 podStartE2EDuration="3.718006329s" totalImagesPullingTime="2.247114486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:20:18.716199326 +0000 UTC m=+395.289296130" watchObservedRunningTime="2026-07-24 00:20:18.718006329 +0000 UTC m=+395.291103134" Jul 24 00:20:18.730396 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:18.730354 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-vx7gf" podStartSLOduration=1.053983209 podStartE2EDuration="2.730339375s" totalImagesPullingTime="1.676356166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:20:18.728935997 +0000 UTC m=+395.302032813" watchObservedRunningTime="2026-07-24 00:20:18.730339375 +0000 UTC m=+395.303436179" Jul 24 00:20:18.948824 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:18.948795 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-59brs" Jul 24 00:20:18.977009 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:18.976917 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgkcj\" (UniqueName: \"kubernetes.io/projected/8eecb5ef-1eb9-4146-8ef3-ec885ab259d5-kube-api-access-rgkcj\") pod \"8eecb5ef-1eb9-4146-8ef3-ec885ab259d5\" (UID: \"8eecb5ef-1eb9-4146-8ef3-ec885ab259d5\") " Jul 24 00:20:18.980037 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:18.980003 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eecb5ef-1eb9-4146-8ef3-ec885ab259d5-kube-api-access-rgkcj" (OuterVolumeSpecName: "kube-api-access-rgkcj") pod "8eecb5ef-1eb9-4146-8ef3-ec885ab259d5" (UID: "8eecb5ef-1eb9-4146-8ef3-ec885ab259d5"). InnerVolumeSpecName "kube-api-access-rgkcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:20:19.078152 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:19.078093 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgkcj\" (UniqueName: \"kubernetes.io/projected/8eecb5ef-1eb9-4146-8ef3-ec885ab259d5-kube-api-access-rgkcj\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:19.710535 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:19.710502 2576 generic.go:356] "Generic (PLEG): container finished" podID="8eecb5ef-1eb9-4146-8ef3-ec885ab259d5" containerID="1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc" exitCode=0 Jul 24 00:20:19.710981 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:19.710540 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-59brs" Jul 24 00:20:19.710981 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:19.710579 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-59brs" event={"ID":"8eecb5ef-1eb9-4146-8ef3-ec885ab259d5","Type":"ContainerDied","Data":"1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc"} Jul 24 00:20:19.710981 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:19.710622 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-59brs" event={"ID":"8eecb5ef-1eb9-4146-8ef3-ec885ab259d5","Type":"ContainerDied","Data":"adfc8f52dc1edde5a504451ca8db03ecf3cb0fbb6a84d95e68f1538a62b8ee88"} Jul 24 00:20:19.710981 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:19.710641 2576 scope.go:117] "RemoveContainer" containerID="1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc" Jul 24 00:20:19.720640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:19.720623 2576 scope.go:117] "RemoveContainer" containerID="1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc" Jul 24 00:20:19.720896 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:20:19.720877 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc\": container with ID starting with 1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc not found: ID does not exist" containerID="1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc" Jul 24 00:20:19.720981 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:19.720903 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc"} err="failed to get container status \"1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc\": rpc error: code = NotFound desc = could not find container \"1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc\": container with ID starting with 1d4810d31ed4a45bd8d622dcda7df512182f229e0140459cb5fa25600c9e11cc not found: ID does not exist" Jul 24 00:20:19.731967 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:19.731895 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-59brs"] Jul 24 00:20:19.736304 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:19.736282 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-59brs"] Jul 24 00:20:20.008540 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:20.008461 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eecb5ef-1eb9-4146-8ef3-ec885ab259d5" path="/var/lib/kubelet/pods/8eecb5ef-1eb9-4146-8ef3-ec885ab259d5/volumes" Jul 24 00:20:26.627061 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:26.627021 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-vx7gf" Jul 24 00:20:26.627499 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:26.627088 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-vx7gf" Jul 24 00:20:26.648485 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:26.648457 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-vx7gf" Jul 24 00:20:26.764647 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:26.764610 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-vx7gf" Jul 24 00:20:28.349037 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.349003 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls"] Jul 24 00:20:28.349429 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.349314 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eecb5ef-1eb9-4146-8ef3-ec885ab259d5" containerName="registry-server" Jul 24 00:20:28.349429 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.349326 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eecb5ef-1eb9-4146-8ef3-ec885ab259d5" containerName="registry-server" Jul 24 00:20:28.349429 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.349378 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eecb5ef-1eb9-4146-8ef3-ec885ab259d5" containerName="registry-server" Jul 24 00:20:28.353773 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.353753 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:28.356024 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.356004 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sbjbq\"" Jul 24 00:20:28.360377 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.360353 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls"] Jul 24 00:20:28.458758 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.458712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crpsd\" (UniqueName: \"kubernetes.io/projected/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-kube-api-access-crpsd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls\" (UID: \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:28.458939 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.458818 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls\" (UID: \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:28.458939 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.458863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls\" (UID: \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:28.560252 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.560214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls\" (UID: \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:28.560440 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.560286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crpsd\" (UniqueName: \"kubernetes.io/projected/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-kube-api-access-crpsd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls\" (UID: \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:28.560440 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.560337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls\" (UID: \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:28.560627 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.560569 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls\" (UID: \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:28.560694 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.560668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls\" (UID: \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:28.568464 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.568441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crpsd\" (UniqueName: \"kubernetes.io/projected/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-kube-api-access-crpsd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls\" (UID: \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:28.664074 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.663990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:28.799692 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.799662 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls"] Jul 24 00:20:28.801655 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:20:28.801628 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c330cf_373a_4b2c_ba21_bd34dc15ce09.slice/crio-a455ac14bc58fd572c2d8f78bbb177b5297bcd6518fad3850dd0314d34910671 WatchSource:0}: Error finding container a455ac14bc58fd572c2d8f78bbb177b5297bcd6518fad3850dd0314d34910671: Status 404 returned error can't find the container with id a455ac14bc58fd572c2d8f78bbb177b5297bcd6518fad3850dd0314d34910671 Jul 24 00:20:28.948841 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.948760 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8"] Jul 24 00:20:28.952116 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.952095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:28.960219 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:28.960187 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8"] Jul 24 00:20:29.064325 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.064266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnxn\" (UniqueName: \"kubernetes.io/projected/e8997ccf-869c-459d-ad1a-bbb82537e59e-kube-api-access-vhnxn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8\" (UID: \"e8997ccf-869c-459d-ad1a-bbb82537e59e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:29.064325 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.064318 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8997ccf-869c-459d-ad1a-bbb82537e59e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8\" (UID: \"e8997ccf-869c-459d-ad1a-bbb82537e59e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:29.064561 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.064435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8997ccf-869c-459d-ad1a-bbb82537e59e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8\" (UID: \"e8997ccf-869c-459d-ad1a-bbb82537e59e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:29.165382 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.165346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8997ccf-869c-459d-ad1a-bbb82537e59e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8\" (UID: \"e8997ccf-869c-459d-ad1a-bbb82537e59e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:29.165521 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.165401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnxn\" (UniqueName: \"kubernetes.io/projected/e8997ccf-869c-459d-ad1a-bbb82537e59e-kube-api-access-vhnxn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8\" (UID: \"e8997ccf-869c-459d-ad1a-bbb82537e59e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:29.165521 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.165430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8997ccf-869c-459d-ad1a-bbb82537e59e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8\" (UID: \"e8997ccf-869c-459d-ad1a-bbb82537e59e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:29.165727 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.165705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8997ccf-869c-459d-ad1a-bbb82537e59e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8\" (UID: \"e8997ccf-869c-459d-ad1a-bbb82537e59e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:29.165793 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.165771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8997ccf-869c-459d-ad1a-bbb82537e59e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8\" (UID: \"e8997ccf-869c-459d-ad1a-bbb82537e59e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:29.173511 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.173487 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnxn\" (UniqueName: \"kubernetes.io/projected/e8997ccf-869c-459d-ad1a-bbb82537e59e-kube-api-access-vhnxn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8\" (UID: \"e8997ccf-869c-459d-ad1a-bbb82537e59e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:29.270178 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.270077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:29.397959 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.397926 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8"] Jul 24 00:20:29.399544 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:20:29.399518 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8997ccf_869c_459d_ad1a_bbb82537e59e.slice/crio-a3439b4ef8cd6c85fb1ed078879819ff78f6882256871da26df43f9bc6aae968 WatchSource:0}: Error finding container a3439b4ef8cd6c85fb1ed078879819ff78f6882256871da26df43f9bc6aae968: Status 404 returned error can't find the container with id a3439b4ef8cd6c85fb1ed078879819ff78f6882256871da26df43f9bc6aae968 Jul 24 00:20:29.557908 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.557866 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn"] Jul 24 00:20:29.561316 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.561298 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:29.569115 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.569082 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn"] Jul 24 00:20:29.669625 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.669587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn\" (UID: \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:29.669625 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.669626 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb2rb\" (UniqueName: \"kubernetes.io/projected/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-kube-api-access-tb2rb\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn\" (UID: \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:29.669836 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.669663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn\" (UID: \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:29.761290 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.761253 2576 generic.go:356] "Generic (PLEG): container finished" podID="c1c330cf-373a-4b2c-ba21-bd34dc15ce09" containerID="0682af08f6c8f44e9c86f53589f1bb569fdc76c65ae174dcb62c3b2d8ed57a21" exitCode=0 Jul 24 00:20:29.761460 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.761339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" event={"ID":"c1c330cf-373a-4b2c-ba21-bd34dc15ce09","Type":"ContainerDied","Data":"0682af08f6c8f44e9c86f53589f1bb569fdc76c65ae174dcb62c3b2d8ed57a21"} Jul 24 00:20:29.761460 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.761389 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" event={"ID":"c1c330cf-373a-4b2c-ba21-bd34dc15ce09","Type":"ContainerStarted","Data":"a455ac14bc58fd572c2d8f78bbb177b5297bcd6518fad3850dd0314d34910671"} Jul 24 00:20:29.763025 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.763003 2576 generic.go:356] "Generic (PLEG): container finished" podID="e8997ccf-869c-459d-ad1a-bbb82537e59e" containerID="45e364e901b3b03694c3733478c58cb2dbc03d149b97c00247dc01bf3e65b67f" exitCode=0 Jul 24 00:20:29.763142 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.763064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" event={"ID":"e8997ccf-869c-459d-ad1a-bbb82537e59e","Type":"ContainerDied","Data":"45e364e901b3b03694c3733478c58cb2dbc03d149b97c00247dc01bf3e65b67f"} Jul 24 00:20:29.763142 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.763096 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" event={"ID":"e8997ccf-869c-459d-ad1a-bbb82537e59e","Type":"ContainerStarted","Data":"a3439b4ef8cd6c85fb1ed078879819ff78f6882256871da26df43f9bc6aae968"} Jul 24 00:20:29.770109 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.770079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn\" (UID: \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:29.770209 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.770120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb2rb\" (UniqueName: \"kubernetes.io/projected/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-kube-api-access-tb2rb\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn\" (UID: \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:29.770209 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.770161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn\" (UID: \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:29.770439 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.770423 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn\" (UID: \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:29.770479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.770465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn\" (UID: \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:29.781071 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.781044 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb2rb\" (UniqueName: \"kubernetes.io/projected/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-kube-api-access-tb2rb\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn\" (UID: \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:29.882182 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:29.882093 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:30.017865 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.017820 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn"] Jul 24 00:20:30.020091 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:20:30.020064 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a1f99d_14d2_4bea_8758_dfbaa40a1172.slice/crio-697cbeef1ebb7e0a3a90699636f2d08503f3fc134cf59d4f27326c2995e6ae07 WatchSource:0}: Error finding container 697cbeef1ebb7e0a3a90699636f2d08503f3fc134cf59d4f27326c2995e6ae07: Status 404 returned error can't find the container with id 697cbeef1ebb7e0a3a90699636f2d08503f3fc134cf59d4f27326c2995e6ae07 Jul 24 00:20:30.147990 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.147913 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk"] Jul 24 00:20:30.154309 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.152622 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:30.158679 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.158654 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk"] Jul 24 00:20:30.274598 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.274564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af711c37-e5c5-4636-a72a-3dbb9d3f0176-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk\" (UID: \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:30.274734 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.274650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af711c37-e5c5-4636-a72a-3dbb9d3f0176-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk\" (UID: \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:30.274734 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.274669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkk5\" (UniqueName: \"kubernetes.io/projected/af711c37-e5c5-4636-a72a-3dbb9d3f0176-kube-api-access-qfkk5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk\" (UID: \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:30.375836 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.375803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af711c37-e5c5-4636-a72a-3dbb9d3f0176-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk\" (UID: \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:30.376033 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.375843 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkk5\" (UniqueName: \"kubernetes.io/projected/af711c37-e5c5-4636-a72a-3dbb9d3f0176-kube-api-access-qfkk5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk\" (UID: \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:30.376033 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.375885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af711c37-e5c5-4636-a72a-3dbb9d3f0176-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk\" (UID: \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:30.376199 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.376179 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af711c37-e5c5-4636-a72a-3dbb9d3f0176-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk\" (UID: \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:30.376291 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.376276 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af711c37-e5c5-4636-a72a-3dbb9d3f0176-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk\" (UID: \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:30.383989 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.383966 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkk5\" (UniqueName: \"kubernetes.io/projected/af711c37-e5c5-4636-a72a-3dbb9d3f0176-kube-api-access-qfkk5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk\" (UID: \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:30.480326 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.480248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:30.772461 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.772199 2576 generic.go:356] "Generic (PLEG): container finished" podID="b0a1f99d-14d2-4bea-8758-dfbaa40a1172" containerID="0b3077dd3ec92699221654b9c0734247dcb134133857f1798ab694594c691450" exitCode=0 Jul 24 00:20:30.772461 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.772249 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" event={"ID":"b0a1f99d-14d2-4bea-8758-dfbaa40a1172","Type":"ContainerDied","Data":"0b3077dd3ec92699221654b9c0734247dcb134133857f1798ab694594c691450"} Jul 24 00:20:30.772461 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.772301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" event={"ID":"b0a1f99d-14d2-4bea-8758-dfbaa40a1172","Type":"ContainerStarted","Data":"697cbeef1ebb7e0a3a90699636f2d08503f3fc134cf59d4f27326c2995e6ae07"} Jul 24 00:20:30.869387 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:30.869363 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk"] Jul 24 00:20:30.914821 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:20:30.914784 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf711c37_e5c5_4636_a72a_3dbb9d3f0176.slice/crio-c5dfab673d3ee685598a5bb4bb4f990941954717e48c70eb750a56249057f2ad WatchSource:0}: Error finding container c5dfab673d3ee685598a5bb4bb4f990941954717e48c70eb750a56249057f2ad: Status 404 returned error can't find the container with id c5dfab673d3ee685598a5bb4bb4f990941954717e48c70eb750a56249057f2ad Jul 24 00:20:31.779297 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:31.779212 2576 generic.go:356] "Generic (PLEG): container finished" podID="e8997ccf-869c-459d-ad1a-bbb82537e59e" containerID="1365b750ff224073c75ba32eb0a0a006522130b580689695e112cd6a3ecf3bbc" exitCode=0 Jul 24 00:20:31.779703 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:31.779293 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" event={"ID":"e8997ccf-869c-459d-ad1a-bbb82537e59e","Type":"ContainerDied","Data":"1365b750ff224073c75ba32eb0a0a006522130b580689695e112cd6a3ecf3bbc"} Jul 24 00:20:31.780874 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:31.780849 2576 generic.go:356] "Generic (PLEG): container finished" podID="af711c37-e5c5-4636-a72a-3dbb9d3f0176" containerID="b01b4997b838feb65a994459f6b0e545458060b8f7fd2e70823de8d5c544ac23" exitCode=0 Jul 24 00:20:31.780970 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:31.780925 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" event={"ID":"af711c37-e5c5-4636-a72a-3dbb9d3f0176","Type":"ContainerDied","Data":"b01b4997b838feb65a994459f6b0e545458060b8f7fd2e70823de8d5c544ac23"} Jul 24 00:20:31.781039 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:31.781001 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" event={"ID":"af711c37-e5c5-4636-a72a-3dbb9d3f0176","Type":"ContainerStarted","Data":"c5dfab673d3ee685598a5bb4bb4f990941954717e48c70eb750a56249057f2ad"} Jul 24 00:20:31.782838 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:31.782812 2576 generic.go:356] "Generic (PLEG): container finished" podID="b0a1f99d-14d2-4bea-8758-dfbaa40a1172" containerID="27200ca983e31eed02e92c0d90dac91e4226e03861af618f32cebcb6ac8c1637" exitCode=0 Jul 24 00:20:31.782937 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:31.782885 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" event={"ID":"b0a1f99d-14d2-4bea-8758-dfbaa40a1172","Type":"ContainerDied","Data":"27200ca983e31eed02e92c0d90dac91e4226e03861af618f32cebcb6ac8c1637"} Jul 24 00:20:31.784943 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:31.784906 2576 generic.go:356] "Generic (PLEG): container finished" podID="c1c330cf-373a-4b2c-ba21-bd34dc15ce09" containerID="0374b2818c0395c0cdb7f96f6c6f76627ae79c6b2ef03dc0e6ca5fcd3eb37a4f" exitCode=0 Jul 24 00:20:31.785012 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:31.784968 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" event={"ID":"c1c330cf-373a-4b2c-ba21-bd34dc15ce09","Type":"ContainerDied","Data":"0374b2818c0395c0cdb7f96f6c6f76627ae79c6b2ef03dc0e6ca5fcd3eb37a4f"} Jul 24 00:20:32.793967 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:32.793919 2576 generic.go:356] "Generic (PLEG): container finished" podID="c1c330cf-373a-4b2c-ba21-bd34dc15ce09" containerID="1b45e608539607272635782872a45e80eee99b5956c584a7f2bb0f542505c009" exitCode=0 Jul 24 00:20:32.794379 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:32.794007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" event={"ID":"c1c330cf-373a-4b2c-ba21-bd34dc15ce09","Type":"ContainerDied","Data":"1b45e608539607272635782872a45e80eee99b5956c584a7f2bb0f542505c009"} Jul 24 00:20:32.795998 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:32.795976 2576 generic.go:356] "Generic (PLEG): container finished" podID="e8997ccf-869c-459d-ad1a-bbb82537e59e" containerID="425ae60eb0a4f9a8aaa4c8c1f06c28f5e9c69b4ebf0955ae6d397179bd4bcd4f" exitCode=0 Jul 24 00:20:32.796123 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:32.796055 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" event={"ID":"e8997ccf-869c-459d-ad1a-bbb82537e59e","Type":"ContainerDied","Data":"425ae60eb0a4f9a8aaa4c8c1f06c28f5e9c69b4ebf0955ae6d397179bd4bcd4f"} Jul 24 00:20:32.797673 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:32.797648 2576 generic.go:356] "Generic (PLEG): container finished" podID="af711c37-e5c5-4636-a72a-3dbb9d3f0176" containerID="873675f212a872ec196802f658fd404aaffcb035ca8585886d79b68c1713c910" exitCode=0 Jul 24 00:20:32.797760 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:32.797687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" event={"ID":"af711c37-e5c5-4636-a72a-3dbb9d3f0176","Type":"ContainerDied","Data":"873675f212a872ec196802f658fd404aaffcb035ca8585886d79b68c1713c910"} Jul 24 00:20:32.799901 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:32.799879 2576 generic.go:356] "Generic (PLEG): container finished" podID="b0a1f99d-14d2-4bea-8758-dfbaa40a1172" containerID="7d8e9427727d0cb17d76bc210a0ed094d9f7c638b3d1ca9faf81bb40cba45f35" exitCode=0 Jul 24 00:20:32.800016 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:32.799974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" event={"ID":"b0a1f99d-14d2-4bea-8758-dfbaa40a1172","Type":"ContainerDied","Data":"7d8e9427727d0cb17d76bc210a0ed094d9f7c638b3d1ca9faf81bb40cba45f35"} Jul 24 00:20:33.811940 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:33.811904 2576 generic.go:356] "Generic (PLEG): container finished" podID="af711c37-e5c5-4636-a72a-3dbb9d3f0176" containerID="96cbcb7e84caceeca075433f184ad56be2de7d177de159bece2f4ded6009f0e9" exitCode=0 Jul 24 00:20:33.812341 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:33.811991 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" event={"ID":"af711c37-e5c5-4636-a72a-3dbb9d3f0176","Type":"ContainerDied","Data":"96cbcb7e84caceeca075433f184ad56be2de7d177de159bece2f4ded6009f0e9"} Jul 24 00:20:33.959067 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:33.959038 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:33.992813 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:33.992793 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:34.009336 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.009308 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:34.111052 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.110964 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-util\") pod \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\" (UID: \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\") " Jul 24 00:20:34.111052 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.111006 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhnxn\" (UniqueName: \"kubernetes.io/projected/e8997ccf-869c-459d-ad1a-bbb82537e59e-kube-api-access-vhnxn\") pod \"e8997ccf-869c-459d-ad1a-bbb82537e59e\" (UID: \"e8997ccf-869c-459d-ad1a-bbb82537e59e\") " Jul 24 00:20:34.111052 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.111034 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8997ccf-869c-459d-ad1a-bbb82537e59e-bundle\") pod \"e8997ccf-869c-459d-ad1a-bbb82537e59e\" (UID: \"e8997ccf-869c-459d-ad1a-bbb82537e59e\") " Jul 24 00:20:34.111348 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.111086 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb2rb\" (UniqueName: \"kubernetes.io/projected/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-kube-api-access-tb2rb\") pod \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\" (UID: \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\") " Jul 24 00:20:34.111348 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.111110 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-util\") pod \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\" (UID: \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\") " Jul 24 00:20:34.111348 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.111154 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8997ccf-869c-459d-ad1a-bbb82537e59e-util\") pod \"e8997ccf-869c-459d-ad1a-bbb82537e59e\" (UID: \"e8997ccf-869c-459d-ad1a-bbb82537e59e\") " Jul 24 00:20:34.111348 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.111206 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-bundle\") pod \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\" (UID: \"b0a1f99d-14d2-4bea-8758-dfbaa40a1172\") " Jul 24 00:20:34.111348 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.111254 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crpsd\" (UniqueName: \"kubernetes.io/projected/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-kube-api-access-crpsd\") pod \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\" (UID: \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\") " Jul 24 00:20:34.111348 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.111283 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-bundle\") pod \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\" (UID: \"c1c330cf-373a-4b2c-ba21-bd34dc15ce09\") " Jul 24 00:20:34.111786 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.111756 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8997ccf-869c-459d-ad1a-bbb82537e59e-bundle" (OuterVolumeSpecName: "bundle") pod "e8997ccf-869c-459d-ad1a-bbb82537e59e" (UID: "e8997ccf-869c-459d-ad1a-bbb82537e59e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:20:34.112488 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.112457 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-bundle" (OuterVolumeSpecName: "bundle") pod "c1c330cf-373a-4b2c-ba21-bd34dc15ce09" (UID: "c1c330cf-373a-4b2c-ba21-bd34dc15ce09"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:20:34.112676 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.112632 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-bundle" (OuterVolumeSpecName: "bundle") pod "b0a1f99d-14d2-4bea-8758-dfbaa40a1172" (UID: "b0a1f99d-14d2-4bea-8758-dfbaa40a1172"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:20:34.113504 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.113478 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-kube-api-access-tb2rb" (OuterVolumeSpecName: "kube-api-access-tb2rb") pod "b0a1f99d-14d2-4bea-8758-dfbaa40a1172" (UID: "b0a1f99d-14d2-4bea-8758-dfbaa40a1172"). InnerVolumeSpecName "kube-api-access-tb2rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:20:34.113662 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.113647 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8997ccf-869c-459d-ad1a-bbb82537e59e-kube-api-access-vhnxn" (OuterVolumeSpecName: "kube-api-access-vhnxn") pod "e8997ccf-869c-459d-ad1a-bbb82537e59e" (UID: "e8997ccf-869c-459d-ad1a-bbb82537e59e"). InnerVolumeSpecName "kube-api-access-vhnxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:20:34.114094 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.114072 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-kube-api-access-crpsd" (OuterVolumeSpecName: "kube-api-access-crpsd") pod "c1c330cf-373a-4b2c-ba21-bd34dc15ce09" (UID: "c1c330cf-373a-4b2c-ba21-bd34dc15ce09"). InnerVolumeSpecName "kube-api-access-crpsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:20:34.116456 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.116427 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-util" (OuterVolumeSpecName: "util") pod "c1c330cf-373a-4b2c-ba21-bd34dc15ce09" (UID: "c1c330cf-373a-4b2c-ba21-bd34dc15ce09"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:20:34.120425 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.120391 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8997ccf-869c-459d-ad1a-bbb82537e59e-util" (OuterVolumeSpecName: "util") pod "e8997ccf-869c-459d-ad1a-bbb82537e59e" (UID: "e8997ccf-869c-459d-ad1a-bbb82537e59e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:20:34.120555 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.120538 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-util" (OuterVolumeSpecName: "util") pod "b0a1f99d-14d2-4bea-8758-dfbaa40a1172" (UID: "b0a1f99d-14d2-4bea-8758-dfbaa40a1172"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:20:34.212421 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.212377 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8997ccf-869c-459d-ad1a-bbb82537e59e-bundle\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:34.212421 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.212415 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb2rb\" (UniqueName: \"kubernetes.io/projected/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-kube-api-access-tb2rb\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:34.212421 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.212425 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-util\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:34.212421 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.212435 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8997ccf-869c-459d-ad1a-bbb82537e59e-util\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:34.212687 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.212443 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0a1f99d-14d2-4bea-8758-dfbaa40a1172-bundle\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:34.212687 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.212451 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crpsd\" (UniqueName: \"kubernetes.io/projected/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-kube-api-access-crpsd\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:34.212687 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.212459 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-bundle\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:34.212687 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.212467 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1c330cf-373a-4b2c-ba21-bd34dc15ce09-util\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:34.212687 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.212474 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhnxn\" (UniqueName: \"kubernetes.io/projected/e8997ccf-869c-459d-ad1a-bbb82537e59e-kube-api-access-vhnxn\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:34.818446 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.818418 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" Jul 24 00:20:34.818863 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.818445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls" event={"ID":"c1c330cf-373a-4b2c-ba21-bd34dc15ce09","Type":"ContainerDied","Data":"a455ac14bc58fd572c2d8f78bbb177b5297bcd6518fad3850dd0314d34910671"} Jul 24 00:20:34.818863 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.818495 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a455ac14bc58fd572c2d8f78bbb177b5297bcd6518fad3850dd0314d34910671" Jul 24 00:20:34.820390 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.820372 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" Jul 24 00:20:34.820519 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.820406 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8" event={"ID":"e8997ccf-869c-459d-ad1a-bbb82537e59e","Type":"ContainerDied","Data":"a3439b4ef8cd6c85fb1ed078879819ff78f6882256871da26df43f9bc6aae968"} Jul 24 00:20:34.820519 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.820442 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3439b4ef8cd6c85fb1ed078879819ff78f6882256871da26df43f9bc6aae968" Jul 24 00:20:34.822244 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.822222 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" Jul 24 00:20:34.822353 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.822262 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn" event={"ID":"b0a1f99d-14d2-4bea-8758-dfbaa40a1172","Type":"ContainerDied","Data":"697cbeef1ebb7e0a3a90699636f2d08503f3fc134cf59d4f27326c2995e6ae07"} Jul 24 00:20:34.822353 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.822303 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697cbeef1ebb7e0a3a90699636f2d08503f3fc134cf59d4f27326c2995e6ae07" Jul 24 00:20:34.940489 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:34.940469 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:35.018154 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:35.018116 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af711c37-e5c5-4636-a72a-3dbb9d3f0176-util\") pod \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\" (UID: \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\") " Jul 24 00:20:35.018349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:35.018216 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af711c37-e5c5-4636-a72a-3dbb9d3f0176-bundle\") pod \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\" (UID: \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\") " Jul 24 00:20:35.018349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:35.018276 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfkk5\" (UniqueName: \"kubernetes.io/projected/af711c37-e5c5-4636-a72a-3dbb9d3f0176-kube-api-access-qfkk5\") pod \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\" (UID: \"af711c37-e5c5-4636-a72a-3dbb9d3f0176\") " Jul 24 00:20:35.018908 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:35.018880 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af711c37-e5c5-4636-a72a-3dbb9d3f0176-bundle" (OuterVolumeSpecName: "bundle") pod "af711c37-e5c5-4636-a72a-3dbb9d3f0176" (UID: "af711c37-e5c5-4636-a72a-3dbb9d3f0176"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:20:35.020394 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:35.020364 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af711c37-e5c5-4636-a72a-3dbb9d3f0176-kube-api-access-qfkk5" (OuterVolumeSpecName: "kube-api-access-qfkk5") pod "af711c37-e5c5-4636-a72a-3dbb9d3f0176" (UID: "af711c37-e5c5-4636-a72a-3dbb9d3f0176"). InnerVolumeSpecName "kube-api-access-qfkk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:20:35.023599 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:35.023573 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af711c37-e5c5-4636-a72a-3dbb9d3f0176-util" (OuterVolumeSpecName: "util") pod "af711c37-e5c5-4636-a72a-3dbb9d3f0176" (UID: "af711c37-e5c5-4636-a72a-3dbb9d3f0176"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:20:35.119136 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:35.119048 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af711c37-e5c5-4636-a72a-3dbb9d3f0176-util\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:35.119136 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:35.119075 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af711c37-e5c5-4636-a72a-3dbb9d3f0176-bundle\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:35.119136 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:35.119086 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfkk5\" (UniqueName: \"kubernetes.io/projected/af711c37-e5c5-4636-a72a-3dbb9d3f0176-kube-api-access-qfkk5\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:20:35.828296 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:35.828269 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" Jul 24 00:20:35.828698 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:35.828269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk" event={"ID":"af711c37-e5c5-4636-a72a-3dbb9d3f0176","Type":"ContainerDied","Data":"c5dfab673d3ee685598a5bb4bb4f990941954717e48c70eb750a56249057f2ad"} Jul 24 00:20:35.828698 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:35.828390 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5dfab673d3ee685598a5bb4bb4f990941954717e48c70eb750a56249057f2ad" Jul 24 00:20:46.511041 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511004 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq"] Jul 24 00:20:46.511442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511354 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a1f99d-14d2-4bea-8758-dfbaa40a1172" containerName="extract" Jul 24 00:20:46.511442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511368 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a1f99d-14d2-4bea-8758-dfbaa40a1172" containerName="extract" Jul 24 00:20:46.511442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511381 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c330cf-373a-4b2c-ba21-bd34dc15ce09" containerName="extract" Jul 24 00:20:46.511442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511386 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c330cf-373a-4b2c-ba21-bd34dc15ce09" containerName="extract" Jul 24 00:20:46.511442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511393 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8997ccf-869c-459d-ad1a-bbb82537e59e" containerName="util" Jul 24 00:20:46.511442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511400 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8997ccf-869c-459d-ad1a-bbb82537e59e" containerName="util" Jul 24 00:20:46.511442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511411 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a1f99d-14d2-4bea-8758-dfbaa40a1172" containerName="util" Jul 24 00:20:46.511442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511417 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a1f99d-14d2-4bea-8758-dfbaa40a1172" containerName="util" Jul 24 00:20:46.511442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511423 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af711c37-e5c5-4636-a72a-3dbb9d3f0176" containerName="util" Jul 24 00:20:46.511442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511430 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="af711c37-e5c5-4636-a72a-3dbb9d3f0176" containerName="util" Jul 24 00:20:46.511442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511436 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af711c37-e5c5-4636-a72a-3dbb9d3f0176" containerName="extract" Jul 24 00:20:46.511442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511441 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="af711c37-e5c5-4636-a72a-3dbb9d3f0176" containerName="extract" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511451 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c330cf-373a-4b2c-ba21-bd34dc15ce09" containerName="util" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511456 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c330cf-373a-4b2c-ba21-bd34dc15ce09" containerName="util" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511464 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8997ccf-869c-459d-ad1a-bbb82537e59e" containerName="pull" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511469 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8997ccf-869c-459d-ad1a-bbb82537e59e" containerName="pull" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511474 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c330cf-373a-4b2c-ba21-bd34dc15ce09" containerName="pull" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511482 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c330cf-373a-4b2c-ba21-bd34dc15ce09" containerName="pull" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511491 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af711c37-e5c5-4636-a72a-3dbb9d3f0176" containerName="pull" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511496 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="af711c37-e5c5-4636-a72a-3dbb9d3f0176" containerName="pull" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511505 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a1f99d-14d2-4bea-8758-dfbaa40a1172" containerName="pull" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511510 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a1f99d-14d2-4bea-8758-dfbaa40a1172" containerName="pull" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511515 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8997ccf-869c-459d-ad1a-bbb82537e59e" containerName="extract" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511520 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8997ccf-869c-459d-ad1a-bbb82537e59e" containerName="extract" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511576 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1c330cf-373a-4b2c-ba21-bd34dc15ce09" containerName="extract" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511583 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8997ccf-869c-459d-ad1a-bbb82537e59e" containerName="extract" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511591 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="af711c37-e5c5-4636-a72a-3dbb9d3f0176" containerName="extract" Jul 24 00:20:46.511859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.511597 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a1f99d-14d2-4bea-8758-dfbaa40a1172" containerName="extract" Jul 24 00:20:46.515594 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.515576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:20:46.518640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.518621 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5xpbk\"" Jul 24 00:20:46.527392 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.527370 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq"] Jul 24 00:20:46.620259 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.620219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmfjl\" (UniqueName: \"kubernetes.io/projected/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e-kube-api-access-fmfjl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-95hfq\" (UID: \"aee0f84c-7afa-4d96-80bf-2fd97cd8da6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:20:46.620440 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.620270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-95hfq\" (UID: \"aee0f84c-7afa-4d96-80bf-2fd97cd8da6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:20:46.721434 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.721402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmfjl\" (UniqueName: \"kubernetes.io/projected/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e-kube-api-access-fmfjl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-95hfq\" (UID: \"aee0f84c-7afa-4d96-80bf-2fd97cd8da6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:20:46.721592 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.721452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-95hfq\" (UID: \"aee0f84c-7afa-4d96-80bf-2fd97cd8da6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:20:46.721802 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.721780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-95hfq\" (UID: \"aee0f84c-7afa-4d96-80bf-2fd97cd8da6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:20:46.732992 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.732966 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmfjl\" (UniqueName: \"kubernetes.io/projected/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e-kube-api-access-fmfjl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-95hfq\" (UID: \"aee0f84c-7afa-4d96-80bf-2fd97cd8da6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:20:46.826128 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.826095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:20:46.966559 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:46.966527 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq"] Jul 24 00:20:46.968116 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:20:46.968086 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee0f84c_7afa_4d96_80bf_2fd97cd8da6e.slice/crio-543084db81f2230f2d9171687e28ab0d8ffd49cf5c1922ed84d394fa90dde2b2 WatchSource:0}: Error finding container 543084db81f2230f2d9171687e28ab0d8ffd49cf5c1922ed84d394fa90dde2b2: Status 404 returned error can't find the container with id 543084db81f2230f2d9171687e28ab0d8ffd49cf5c1922ed84d394fa90dde2b2 Jul 24 00:20:47.890615 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:47.890584 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" event={"ID":"aee0f84c-7afa-4d96-80bf-2fd97cd8da6e","Type":"ContainerStarted","Data":"543084db81f2230f2d9171687e28ab0d8ffd49cf5c1922ed84d394fa90dde2b2"} Jul 24 00:20:51.405890 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:51.405847 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz"] Jul 24 00:20:51.409658 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:51.409636 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz" Jul 24 00:20:51.412137 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:51.412113 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 00:20:51.412263 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:51.412163 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pclnn\"" Jul 24 00:20:51.424551 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:51.424527 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz"] Jul 24 00:20:51.564709 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:51.564669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsf2b\" (UniqueName: \"kubernetes.io/projected/5cd36eda-9782-4c23-a860-f6a277925add-kube-api-access-vsf2b\") pod \"dns-operator-controller-manager-864dd78c4c-t8htz\" (UID: \"5cd36eda-9782-4c23-a860-f6a277925add\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz" Jul 24 00:20:51.665424 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:51.665340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsf2b\" (UniqueName: \"kubernetes.io/projected/5cd36eda-9782-4c23-a860-f6a277925add-kube-api-access-vsf2b\") pod \"dns-operator-controller-manager-864dd78c4c-t8htz\" (UID: \"5cd36eda-9782-4c23-a860-f6a277925add\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz" Jul 24 00:20:51.676793 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:51.676760 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsf2b\" (UniqueName: \"kubernetes.io/projected/5cd36eda-9782-4c23-a860-f6a277925add-kube-api-access-vsf2b\") pod \"dns-operator-controller-manager-864dd78c4c-t8htz\" (UID: \"5cd36eda-9782-4c23-a860-f6a277925add\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz" Jul 24 00:20:51.722661 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:51.722622 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz" Jul 24 00:20:51.869271 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:51.869239 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz"] Jul 24 00:20:51.872499 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:20:51.872469 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd36eda_9782_4c23_a860_f6a277925add.slice/crio-a9a708b765393665ae4d1c95732562ae53233c31a8e4b2e6a348d0dc4c268d8a WatchSource:0}: Error finding container a9a708b765393665ae4d1c95732562ae53233c31a8e4b2e6a348d0dc4c268d8a: Status 404 returned error can't find the container with id a9a708b765393665ae4d1c95732562ae53233c31a8e4b2e6a348d0dc4c268d8a Jul 24 00:20:51.913699 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:51.913646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz" event={"ID":"5cd36eda-9782-4c23-a860-f6a277925add","Type":"ContainerStarted","Data":"a9a708b765393665ae4d1c95732562ae53233c31a8e4b2e6a348d0dc4c268d8a"} Jul 24 00:20:53.393624 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:53.393589 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk"] Jul 24 00:20:53.397400 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:53.397373 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:20:53.399590 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:53.399567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wkpht\"" Jul 24 00:20:53.408299 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:53.408229 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk"] Jul 24 00:20:53.582236 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:53.582214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65bv4\" (UniqueName: \"kubernetes.io/projected/8c1f2553-0496-4507-9e30-f4c13d7bdfff-kube-api-access-65bv4\") pod \"limitador-operator-controller-manager-859d7b78bd-vmhgk\" (UID: \"8c1f2553-0496-4507-9e30-f4c13d7bdfff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:20:53.682785 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:53.682709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bv4\" (UniqueName: \"kubernetes.io/projected/8c1f2553-0496-4507-9e30-f4c13d7bdfff-kube-api-access-65bv4\") pod \"limitador-operator-controller-manager-859d7b78bd-vmhgk\" (UID: \"8c1f2553-0496-4507-9e30-f4c13d7bdfff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:20:53.691076 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:53.691049 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65bv4\" (UniqueName: \"kubernetes.io/projected/8c1f2553-0496-4507-9e30-f4c13d7bdfff-kube-api-access-65bv4\") pod \"limitador-operator-controller-manager-859d7b78bd-vmhgk\" (UID: \"8c1f2553-0496-4507-9e30-f4c13d7bdfff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:20:53.712435 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:53.712401 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:20:53.932705 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:53.932650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" event={"ID":"aee0f84c-7afa-4d96-80bf-2fd97cd8da6e","Type":"ContainerStarted","Data":"8e31b268a3f4312fe5e4aca17e6eee51d71d75ce20b8cbd74efb509ddc8f869a"} Jul 24 00:20:53.932912 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:53.932753 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:20:53.946506 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:53.946482 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk"] Jul 24 00:20:53.953785 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:53.953735 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" podStartSLOduration=1.357510318 podStartE2EDuration="7.9537168s" totalImagesPullingTime="6.596206482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:20:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:20:53.951426565 +0000 UTC m=+430.524523382" watchObservedRunningTime="2026-07-24 00:20:53.9537168 +0000 UTC m=+430.526813604" Jul 24 00:20:54.063427 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:20:54.063392 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1f2553_0496_4507_9e30_f4c13d7bdfff.slice/crio-17d0efe1d4d7fb94c775c0ebace0daf06d9dc9fd49bb3c2ce62769e59644bd49 WatchSource:0}: Error finding container 17d0efe1d4d7fb94c775c0ebace0daf06d9dc9fd49bb3c2ce62769e59644bd49: Status 404 returned error can't find the container with id 17d0efe1d4d7fb94c775c0ebace0daf06d9dc9fd49bb3c2ce62769e59644bd49 Jul 24 00:20:54.941178 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:54.941139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" event={"ID":"8c1f2553-0496-4507-9e30-f4c13d7bdfff","Type":"ContainerStarted","Data":"17d0efe1d4d7fb94c775c0ebace0daf06d9dc9fd49bb3c2ce62769e59644bd49"} Jul 24 00:20:55.948727 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:55.948588 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" event={"ID":"8c1f2553-0496-4507-9e30-f4c13d7bdfff","Type":"ContainerStarted","Data":"6a23b8c66d1f04e65dcf0448ba75bca8a36ba241191cfc6995c1b449fc1c4875"} Jul 24 00:20:55.949178 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:55.948791 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:20:55.950542 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:55.950516 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz" event={"ID":"5cd36eda-9782-4c23-a860-f6a277925add","Type":"ContainerStarted","Data":"3872fa819efad0bf401fa6f8786aa73f9c8cacffa64bd6f701056bdce31a3d3a"} Jul 24 00:20:55.950668 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:55.950568 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz" Jul 24 00:20:55.968697 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:55.968652 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" podStartSLOduration=1.307036413 podStartE2EDuration="2.968636556s" totalImagesPullingTime="1.661600143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:20:55.968049477 +0000 UTC m=+432.541146281" watchObservedRunningTime="2026-07-24 00:20:55.968636556 +0000 UTC m=+432.541733356" Jul 24 00:20:55.994216 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:20:55.994169 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz" podStartSLOduration=1.186501789 podStartE2EDuration="4.99415493s" totalImagesPullingTime="3.807653141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:20:55.991871721 +0000 UTC m=+432.564968524" watchObservedRunningTime="2026-07-24 00:20:55.99415493 +0000 UTC m=+432.567251732" Jul 24 00:21:04.944473 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:04.944445 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:21:06.912112 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.912026 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq"] Jul 24 00:21:06.912482 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.912279 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" podUID="aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" containerName="manager" containerID="cri-o://8e31b268a3f4312fe5e4aca17e6eee51d71d75ce20b8cbd74efb509ddc8f869a" gracePeriod=2 Jul 24 00:21:06.918599 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.918553 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq"] Jul 24 00:21:06.926559 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.926490 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk"] Jul 24 00:21:06.927051 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.926994 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" podUID="8c1f2553-0496-4507-9e30-f4c13d7bdfff" containerName="manager" containerID="cri-o://6a23b8c66d1f04e65dcf0448ba75bca8a36ba241191cfc6995c1b449fc1c4875" gracePeriod=2 Jul 24 00:21:06.928820 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:06.928760 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-95hfq\" is forbidden: User \"system:node:ip-10-0-134-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-162.ec2.internal' and this object" podUID="aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:21:06.929221 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.929197 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:21:06.941017 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.940989 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4"] Jul 24 00:21:06.941500 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.941483 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" containerName="manager" Jul 24 00:21:06.941583 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.941505 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" containerName="manager" Jul 24 00:21:06.941634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.941599 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" containerName="manager" Jul 24 00:21:06.944972 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.944927 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk"] Jul 24 00:21:06.945102 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.945078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" Jul 24 00:21:06.947172 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:06.947121 2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"8c1f2553-0496-4507-9e30-f4c13d7bdfff\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-vmhgk\": pods \"limitador-operator-controller-manager-859d7b78bd-vmhgk\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:21:06.950356 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.950333 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw"] Jul 24 00:21:06.950634 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.950623 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c1f2553-0496-4507-9e30-f4c13d7bdfff" containerName="manager" Jul 24 00:21:06.950681 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.950636 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c1f2553-0496-4507-9e30-f4c13d7bdfff" containerName="manager" Jul 24 00:21:06.950718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.950701 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c1f2553-0496-4507-9e30-f4c13d7bdfff" containerName="manager" Jul 24 00:21:06.953471 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.953451 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw" Jul 24 00:21:06.955849 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.955826 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4"] Jul 24 00:21:06.957852 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.957830 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t8htz" Jul 24 00:21:06.968783 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.968762 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw"] Jul 24 00:21:06.974838 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:06.974805 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-95hfq\" is forbidden: User \"system:node:ip-10-0-134-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-162.ec2.internal' and this object" podUID="aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:21:06.977131 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:06.977103 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vmhgk\" is forbidden: User \"system:node:ip-10-0-134-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-162.ec2.internal' and this object" podUID="8c1f2553-0496-4507-9e30-f4c13d7bdfff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:21:06.979232 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:06.979206 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-95hfq\" is forbidden: User \"system:node:ip-10-0-134-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-162.ec2.internal' and this object" podUID="aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:21:06.981165 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:06.981142 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vmhgk\" is forbidden: User \"system:node:ip-10-0-134-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-162.ec2.internal' and this object" podUID="8c1f2553-0496-4507-9e30-f4c13d7bdfff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:21:06.995087 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.995067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pddw6\" (UniqueName: \"kubernetes.io/projected/a9eb28d4-95d2-404d-ae10-2c4354213666-kube-api-access-pddw6\") pod \"limitador-operator-controller-manager-859d7b78bd-8rqgw\" (UID: \"a9eb28d4-95d2-404d-ae10-2c4354213666\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw" Jul 24 00:21:06.995362 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.995103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edd6c941-8f54-47be-ad68-b6696c024cfc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fs4c4\" (UID: \"edd6c941-8f54-47be-ad68-b6696c024cfc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" Jul 24 00:21:06.995362 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:06.995170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghkx9\" (UniqueName: \"kubernetes.io/projected/edd6c941-8f54-47be-ad68-b6696c024cfc-kube-api-access-ghkx9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fs4c4\" (UID: \"edd6c941-8f54-47be-ad68-b6696c024cfc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" Jul 24 00:21:07.096066 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.096029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkx9\" (UniqueName: \"kubernetes.io/projected/edd6c941-8f54-47be-ad68-b6696c024cfc-kube-api-access-ghkx9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fs4c4\" (UID: \"edd6c941-8f54-47be-ad68-b6696c024cfc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" Jul 24 00:21:07.096231 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.096151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pddw6\" (UniqueName: \"kubernetes.io/projected/a9eb28d4-95d2-404d-ae10-2c4354213666-kube-api-access-pddw6\") pod \"limitador-operator-controller-manager-859d7b78bd-8rqgw\" (UID: \"a9eb28d4-95d2-404d-ae10-2c4354213666\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw" Jul 24 00:21:07.096231 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.096192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edd6c941-8f54-47be-ad68-b6696c024cfc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fs4c4\" (UID: \"edd6c941-8f54-47be-ad68-b6696c024cfc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" Jul 24 00:21:07.096781 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.096744 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edd6c941-8f54-47be-ad68-b6696c024cfc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fs4c4\" (UID: \"edd6c941-8f54-47be-ad68-b6696c024cfc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" Jul 24 00:21:07.105523 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.105492 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghkx9\" (UniqueName: \"kubernetes.io/projected/edd6c941-8f54-47be-ad68-b6696c024cfc-kube-api-access-ghkx9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fs4c4\" (UID: \"edd6c941-8f54-47be-ad68-b6696c024cfc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" Jul 24 00:21:07.105675 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.105501 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pddw6\" (UniqueName: \"kubernetes.io/projected/a9eb28d4-95d2-404d-ae10-2c4354213666-kube-api-access-pddw6\") pod \"limitador-operator-controller-manager-859d7b78bd-8rqgw\" (UID: \"a9eb28d4-95d2-404d-ae10-2c4354213666\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw" Jul 24 00:21:07.174096 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.174072 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:21:07.176289 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:07.176260 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-95hfq\" is forbidden: User \"system:node:ip-10-0-134-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-162.ec2.internal' and this object" podUID="aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:21:07.178042 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.178021 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:21:07.178285 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:07.178258 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vmhgk\" is forbidden: User \"system:node:ip-10-0-134-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-162.ec2.internal' and this object" podUID="8c1f2553-0496-4507-9e30-f4c13d7bdfff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:21:07.180055 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:07.180029 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-95hfq\" is forbidden: User \"system:node:ip-10-0-134-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-162.ec2.internal' and this object" podUID="aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:21:07.182057 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:07.182034 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vmhgk\" is forbidden: User \"system:node:ip-10-0-134-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-162.ec2.internal' and this object" podUID="8c1f2553-0496-4507-9e30-f4c13d7bdfff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:21:07.197442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.197413 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e-extensions-socket-volume\") pod \"aee0f84c-7afa-4d96-80bf-2fd97cd8da6e\" (UID: \"aee0f84c-7afa-4d96-80bf-2fd97cd8da6e\") " Jul 24 00:21:07.197558 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.197468 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmfjl\" (UniqueName: \"kubernetes.io/projected/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e-kube-api-access-fmfjl\") pod \"aee0f84c-7afa-4d96-80bf-2fd97cd8da6e\" (UID: \"aee0f84c-7afa-4d96-80bf-2fd97cd8da6e\") " Jul 24 00:21:07.197912 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.197884 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" (UID: "aee0f84c-7afa-4d96-80bf-2fd97cd8da6e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:21:07.199866 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.199840 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e-kube-api-access-fmfjl" (OuterVolumeSpecName: "kube-api-access-fmfjl") pod "aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" (UID: "aee0f84c-7afa-4d96-80bf-2fd97cd8da6e"). InnerVolumeSpecName "kube-api-access-fmfjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:21:07.297978 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.297915 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65bv4\" (UniqueName: \"kubernetes.io/projected/8c1f2553-0496-4507-9e30-f4c13d7bdfff-kube-api-access-65bv4\") pod \"8c1f2553-0496-4507-9e30-f4c13d7bdfff\" (UID: \"8c1f2553-0496-4507-9e30-f4c13d7bdfff\") " Jul 24 00:21:07.298302 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.298282 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e-extensions-socket-volume\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:21:07.298373 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.298306 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmfjl\" (UniqueName: \"kubernetes.io/projected/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e-kube-api-access-fmfjl\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:21:07.300001 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.299977 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1f2553-0496-4507-9e30-f4c13d7bdfff-kube-api-access-65bv4" (OuterVolumeSpecName: "kube-api-access-65bv4") pod "8c1f2553-0496-4507-9e30-f4c13d7bdfff" (UID: "8c1f2553-0496-4507-9e30-f4c13d7bdfff"). InnerVolumeSpecName "kube-api-access-65bv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:21:07.355599 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.355571 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" Jul 24 00:21:07.362745 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.362727 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw" Jul 24 00:21:07.399021 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.398988 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65bv4\" (UniqueName: \"kubernetes.io/projected/8c1f2553-0496-4507-9e30-f4c13d7bdfff-kube-api-access-65bv4\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:21:07.514647 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.514619 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4"] Jul 24 00:21:07.516656 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:21:07.516630 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd6c941_8f54_47be_ad68_b6696c024cfc.slice/crio-6ece122fcb016a693b52156429fba79d584fd4f9e3a37a8e2920b03d6a14e2d3 WatchSource:0}: Error finding container 6ece122fcb016a693b52156429fba79d584fd4f9e3a37a8e2920b03d6a14e2d3: Status 404 returned error can't find the container with id 6ece122fcb016a693b52156429fba79d584fd4f9e3a37a8e2920b03d6a14e2d3 Jul 24 00:21:07.529348 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:07.529324 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw"] Jul 24 00:21:07.530995 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:21:07.530940 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9eb28d4_95d2_404d_ae10_2c4354213666.slice/crio-dead556544f26f09df1c6b405626df24b3fe2f05d2d166f26b3e51bf196f5644 WatchSource:0}: Error finding container dead556544f26f09df1c6b405626df24b3fe2f05d2d166f26b3e51bf196f5644: Status 404 returned error can't find the container with id dead556544f26f09df1c6b405626df24b3fe2f05d2d166f26b3e51bf196f5644 Jul 24 00:21:08.001911 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.001879 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c1f2553-0496-4507-9e30-f4c13d7bdfff" path="/var/lib/kubelet/pods/8c1f2553-0496-4507-9e30-f4c13d7bdfff/volumes" Jul 24 00:21:08.002339 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.002299 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" path="/var/lib/kubelet/pods/aee0f84c-7afa-4d96-80bf-2fd97cd8da6e/volumes" Jul 24 00:21:08.019147 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.019110 2576 generic.go:356] "Generic (PLEG): container finished" podID="aee0f84c-7afa-4d96-80bf-2fd97cd8da6e" containerID="8e31b268a3f4312fe5e4aca17e6eee51d71d75ce20b8cbd74efb509ddc8f869a" exitCode=0 Jul 24 00:21:08.019276 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.019155 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-95hfq" Jul 24 00:21:08.019276 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.019207 2576 scope.go:117] "RemoveContainer" containerID="8e31b268a3f4312fe5e4aca17e6eee51d71d75ce20b8cbd74efb509ddc8f869a" Jul 24 00:21:08.021182 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.021155 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" event={"ID":"edd6c941-8f54-47be-ad68-b6696c024cfc","Type":"ContainerStarted","Data":"f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0"} Jul 24 00:21:08.021363 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.021215 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" Jul 24 00:21:08.021363 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.021241 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" event={"ID":"edd6c941-8f54-47be-ad68-b6696c024cfc","Type":"ContainerStarted","Data":"6ece122fcb016a693b52156429fba79d584fd4f9e3a37a8e2920b03d6a14e2d3"} Jul 24 00:21:08.022513 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.022494 2576 generic.go:356] "Generic (PLEG): container finished" podID="8c1f2553-0496-4507-9e30-f4c13d7bdfff" containerID="6a23b8c66d1f04e65dcf0448ba75bca8a36ba241191cfc6995c1b449fc1c4875" exitCode=0 Jul 24 00:21:08.022594 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.022541 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vmhgk" Jul 24 00:21:08.024233 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.024210 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw" event={"ID":"a9eb28d4-95d2-404d-ae10-2c4354213666","Type":"ContainerStarted","Data":"09a32aadf0300c86860925da05e5ad5be2753cfe657f73298aa34828025a7ee4"} Jul 24 00:21:08.024337 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.024256 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw" event={"ID":"a9eb28d4-95d2-404d-ae10-2c4354213666","Type":"ContainerStarted","Data":"dead556544f26f09df1c6b405626df24b3fe2f05d2d166f26b3e51bf196f5644"} Jul 24 00:21:08.024395 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.024367 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw" Jul 24 00:21:08.029699 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.029681 2576 scope.go:117] "RemoveContainer" containerID="8e31b268a3f4312fe5e4aca17e6eee51d71d75ce20b8cbd74efb509ddc8f869a" Jul 24 00:21:08.029932 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:08.029915 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e31b268a3f4312fe5e4aca17e6eee51d71d75ce20b8cbd74efb509ddc8f869a\": container with ID starting with 8e31b268a3f4312fe5e4aca17e6eee51d71d75ce20b8cbd74efb509ddc8f869a not found: ID does not exist" containerID="8e31b268a3f4312fe5e4aca17e6eee51d71d75ce20b8cbd74efb509ddc8f869a" Jul 24 00:21:08.030006 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.029940 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e31b268a3f4312fe5e4aca17e6eee51d71d75ce20b8cbd74efb509ddc8f869a"} err="failed to get container status \"8e31b268a3f4312fe5e4aca17e6eee51d71d75ce20b8cbd74efb509ddc8f869a\": rpc error: code = NotFound desc = could not find container \"8e31b268a3f4312fe5e4aca17e6eee51d71d75ce20b8cbd74efb509ddc8f869a\": container with ID starting with 8e31b268a3f4312fe5e4aca17e6eee51d71d75ce20b8cbd74efb509ddc8f869a not found: ID does not exist" Jul 24 00:21:08.030006 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.029980 2576 scope.go:117] "RemoveContainer" containerID="6a23b8c66d1f04e65dcf0448ba75bca8a36ba241191cfc6995c1b449fc1c4875" Jul 24 00:21:08.039117 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.039100 2576 scope.go:117] "RemoveContainer" containerID="6a23b8c66d1f04e65dcf0448ba75bca8a36ba241191cfc6995c1b449fc1c4875" Jul 24 00:21:08.039435 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:08.039411 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a23b8c66d1f04e65dcf0448ba75bca8a36ba241191cfc6995c1b449fc1c4875\": container with ID starting with 6a23b8c66d1f04e65dcf0448ba75bca8a36ba241191cfc6995c1b449fc1c4875 not found: ID does not exist" containerID="6a23b8c66d1f04e65dcf0448ba75bca8a36ba241191cfc6995c1b449fc1c4875" Jul 24 00:21:08.039489 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.039445 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a23b8c66d1f04e65dcf0448ba75bca8a36ba241191cfc6995c1b449fc1c4875"} err="failed to get container status \"6a23b8c66d1f04e65dcf0448ba75bca8a36ba241191cfc6995c1b449fc1c4875\": rpc error: code = NotFound desc = could not find container \"6a23b8c66d1f04e65dcf0448ba75bca8a36ba241191cfc6995c1b449fc1c4875\": container with ID starting with 6a23b8c66d1f04e65dcf0448ba75bca8a36ba241191cfc6995c1b449fc1c4875 not found: ID does not exist" Jul 24 00:21:08.044136 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.044098 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" podStartSLOduration=2.044082873 podStartE2EDuration="2.044082873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:21:08.042492761 +0000 UTC m=+444.615589564" watchObservedRunningTime="2026-07-24 00:21:08.044082873 +0000 UTC m=+444.617179676" Jul 24 00:21:08.060423 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:08.060378 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw" podStartSLOduration=2.060367209 podStartE2EDuration="2.060367209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:21:08.058254875 +0000 UTC m=+444.631351684" watchObservedRunningTime="2026-07-24 00:21:08.060367209 +0000 UTC m=+444.633464011" Jul 24 00:21:19.033387 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:19.033352 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8rqgw" Jul 24 00:21:19.033781 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:19.033425 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" Jul 24 00:21:37.713573 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:37.713533 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4"] Jul 24 00:21:37.714166 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:37.713773 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" podUID="edd6c941-8f54-47be-ad68-b6696c024cfc" containerName="manager" containerID="cri-o://f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0" gracePeriod=10 Jul 24 00:21:37.968161 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:37.968096 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" Jul 24 00:21:37.976640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:37.976613 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghkx9\" (UniqueName: \"kubernetes.io/projected/edd6c941-8f54-47be-ad68-b6696c024cfc-kube-api-access-ghkx9\") pod \"edd6c941-8f54-47be-ad68-b6696c024cfc\" (UID: \"edd6c941-8f54-47be-ad68-b6696c024cfc\") " Jul 24 00:21:37.976753 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:37.976691 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edd6c941-8f54-47be-ad68-b6696c024cfc-extensions-socket-volume\") pod \"edd6c941-8f54-47be-ad68-b6696c024cfc\" (UID: \"edd6c941-8f54-47be-ad68-b6696c024cfc\") " Jul 24 00:21:37.977082 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:37.977062 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd6c941-8f54-47be-ad68-b6696c024cfc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "edd6c941-8f54-47be-ad68-b6696c024cfc" (UID: "edd6c941-8f54-47be-ad68-b6696c024cfc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:21:37.978773 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:37.978748 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd6c941-8f54-47be-ad68-b6696c024cfc-kube-api-access-ghkx9" (OuterVolumeSpecName: "kube-api-access-ghkx9") pod "edd6c941-8f54-47be-ad68-b6696c024cfc" (UID: "edd6c941-8f54-47be-ad68-b6696c024cfc"). InnerVolumeSpecName "kube-api-access-ghkx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:21:38.077325 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:38.077282 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghkx9\" (UniqueName: \"kubernetes.io/projected/edd6c941-8f54-47be-ad68-b6696c024cfc-kube-api-access-ghkx9\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:21:38.077325 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:38.077319 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edd6c941-8f54-47be-ad68-b6696c024cfc-extensions-socket-volume\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:21:38.192630 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:38.192594 2576 generic.go:356] "Generic (PLEG): container finished" podID="edd6c941-8f54-47be-ad68-b6696c024cfc" containerID="f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0" exitCode=0 Jul 24 00:21:38.192801 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:38.192640 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" Jul 24 00:21:38.192801 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:38.192682 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" event={"ID":"edd6c941-8f54-47be-ad68-b6696c024cfc","Type":"ContainerDied","Data":"f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0"} Jul 24 00:21:38.192801 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:38.192734 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4" event={"ID":"edd6c941-8f54-47be-ad68-b6696c024cfc","Type":"ContainerDied","Data":"6ece122fcb016a693b52156429fba79d584fd4f9e3a37a8e2920b03d6a14e2d3"} Jul 24 00:21:38.192801 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:38.192756 2576 scope.go:117] "RemoveContainer" containerID="f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0" Jul 24 00:21:38.201666 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:38.201645 2576 scope.go:117] "RemoveContainer" containerID="f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0" Jul 24 00:21:38.201973 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:21:38.201935 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0\": container with ID starting with f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0 not found: ID does not exist" containerID="f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0" Jul 24 00:21:38.202077 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:38.201981 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0"} err="failed to get container status \"f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0\": rpc error: code = NotFound desc = could not find container \"f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0\": container with ID starting with f930489783ae6cb8d0fc2020af2005dfbd0b2613e341804b8103104cc3d23cc0 not found: ID does not exist" Jul 24 00:21:38.211929 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:38.211832 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4"] Jul 24 00:21:38.214478 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:38.214452 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fs4c4"] Jul 24 00:21:40.001593 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:40.001557 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edd6c941-8f54-47be-ad68-b6696c024cfc" path="/var/lib/kubelet/pods/edd6c941-8f54-47be-ad68-b6696c024cfc/volumes" Jul 24 00:21:54.100573 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.100538 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh"] Jul 24 00:21:54.100974 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.100861 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd6c941-8f54-47be-ad68-b6696c024cfc" containerName="manager" Jul 24 00:21:54.100974 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.100871 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd6c941-8f54-47be-ad68-b6696c024cfc" containerName="manager" Jul 24 00:21:54.100974 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.100934 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd6c941-8f54-47be-ad68-b6696c024cfc" containerName="manager" Jul 24 00:21:54.105329 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.105311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.111760 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.111743 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pskc6\"" Jul 24 00:21:54.127110 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.127060 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh"] Jul 24 00:21:54.207410 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.207376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa54590a-51b8-4e30-b65c-55089f26bf3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.207410 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.207415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhqg\" (UniqueName: \"kubernetes.io/projected/fa54590a-51b8-4e30-b65c-55089f26bf3c-kube-api-access-7dhqg\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.207651 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.207453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.207651 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.207470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa54590a-51b8-4e30-b65c-55089f26bf3c-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.207651 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.207526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.207651 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.207559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.207651 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.207583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.207651 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.207624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.207849 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.207660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa54590a-51b8-4e30-b65c-55089f26bf3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.308606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.308568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.308606 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.308607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa54590a-51b8-4e30-b65c-55089f26bf3c-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.308847 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.308636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.308847 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.308660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.308847 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.308693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.308847 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.308727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.308847 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.308765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa54590a-51b8-4e30-b65c-55089f26bf3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.309116 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.308986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa54590a-51b8-4e30-b65c-55089f26bf3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.309116 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.309032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhqg\" (UniqueName: \"kubernetes.io/projected/fa54590a-51b8-4e30-b65c-55089f26bf3c-kube-api-access-7dhqg\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.309116 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.309068 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.309264 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.309134 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.309264 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.309168 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.309264 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.309243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.309563 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.309541 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa54590a-51b8-4e30-b65c-55089f26bf3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.311215 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.311195 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa54590a-51b8-4e30-b65c-55089f26bf3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.311426 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.311410 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa54590a-51b8-4e30-b65c-55089f26bf3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.316821 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.316789 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa54590a-51b8-4e30-b65c-55089f26bf3c-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.317010 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.316990 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhqg\" (UniqueName: \"kubernetes.io/projected/fa54590a-51b8-4e30-b65c-55089f26bf3c-kube-api-access-7dhqg\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rfkdh\" (UID: \"fa54590a-51b8-4e30-b65c-55089f26bf3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.415202 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.415097 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:54.552933 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.552908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh"] Jul 24 00:21:54.554541 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:21:54.554508 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa54590a_51b8_4e30_b65c_55089f26bf3c.slice/crio-8a95b30a4cb2d94e9e68a48f89bc6aeaf41e10a3693b02482fe006c4df692e12 WatchSource:0}: Error finding container 8a95b30a4cb2d94e9e68a48f89bc6aeaf41e10a3693b02482fe006c4df692e12: Status 404 returned error can't find the container with id 8a95b30a4cb2d94e9e68a48f89bc6aeaf41e10a3693b02482fe006c4df692e12 Jul 24 00:21:54.556666 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.556631 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:21:54.556764 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.556747 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:21:54.556810 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:54.556795 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:21:55.284139 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:55.284100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" event={"ID":"fa54590a-51b8-4e30-b65c-55089f26bf3c","Type":"ContainerStarted","Data":"3722609907ea14d1f9e42c850f7711c48edf566c7270c887a1e62b4bd078380d"} Jul 24 00:21:55.284609 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:55.284152 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" event={"ID":"fa54590a-51b8-4e30-b65c-55089f26bf3c","Type":"ContainerStarted","Data":"8a95b30a4cb2d94e9e68a48f89bc6aeaf41e10a3693b02482fe006c4df692e12"} Jul 24 00:21:55.303885 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:55.303826 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" podStartSLOduration=1.303811354 podStartE2EDuration="1.303811354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:21:55.302450413 +0000 UTC m=+491.875547217" watchObservedRunningTime="2026-07-24 00:21:55.303811354 +0000 UTC m=+491.876908157" Jul 24 00:21:55.415447 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:55.415410 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:55.421209 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:55.421178 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:56.290958 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:56.290924 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:56.292002 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:56.291983 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:21:56.800676 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:56.800635 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:21:56.804380 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:56.804347 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" Jul 24 00:21:56.806480 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:56.806457 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sbjbq\"" Jul 24 00:21:56.806599 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:56.806509 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 00:21:56.813248 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:56.813221 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:21:56.841619 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:56.841585 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:21:56.934312 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:56.934273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6lw\" (UniqueName: \"kubernetes.io/projected/d10a100f-6ad4-4ea9-8a30-099f3dd405f8-kube-api-access-px6lw\") pod \"limitador-limitador-54d6d988d5-47jp2\" (UID: \"d10a100f-6ad4-4ea9-8a30-099f3dd405f8\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" Jul 24 00:21:56.934505 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:56.934360 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d10a100f-6ad4-4ea9-8a30-099f3dd405f8-config-file\") pod \"limitador-limitador-54d6d988d5-47jp2\" (UID: \"d10a100f-6ad4-4ea9-8a30-099f3dd405f8\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" Jul 24 00:21:57.035727 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:57.035683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d10a100f-6ad4-4ea9-8a30-099f3dd405f8-config-file\") pod \"limitador-limitador-54d6d988d5-47jp2\" (UID: \"d10a100f-6ad4-4ea9-8a30-099f3dd405f8\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" Jul 24 00:21:57.035929 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:57.035788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6lw\" (UniqueName: \"kubernetes.io/projected/d10a100f-6ad4-4ea9-8a30-099f3dd405f8-kube-api-access-px6lw\") pod \"limitador-limitador-54d6d988d5-47jp2\" (UID: \"d10a100f-6ad4-4ea9-8a30-099f3dd405f8\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" Jul 24 00:21:57.036454 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:57.036432 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d10a100f-6ad4-4ea9-8a30-099f3dd405f8-config-file\") pod \"limitador-limitador-54d6d988d5-47jp2\" (UID: \"d10a100f-6ad4-4ea9-8a30-099f3dd405f8\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" Jul 24 00:21:57.044181 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:57.044160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6lw\" (UniqueName: \"kubernetes.io/projected/d10a100f-6ad4-4ea9-8a30-099f3dd405f8-kube-api-access-px6lw\") pod \"limitador-limitador-54d6d988d5-47jp2\" (UID: \"d10a100f-6ad4-4ea9-8a30-099f3dd405f8\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" Jul 24 00:21:57.116267 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:57.116169 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" Jul 24 00:21:57.263808 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:57.263768 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:21:57.266752 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:21:57.266720 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10a100f_6ad4_4ea9_8a30_099f3dd405f8.slice/crio-4f047fbdba43d6f32d577681501d3e0e8c9db2c0cbe972a85fe67c8e36109b5e WatchSource:0}: Error finding container 4f047fbdba43d6f32d577681501d3e0e8c9db2c0cbe972a85fe67c8e36109b5e: Status 404 returned error can't find the container with id 4f047fbdba43d6f32d577681501d3e0e8c9db2c0cbe972a85fe67c8e36109b5e Jul 24 00:21:57.297387 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:21:57.297347 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" event={"ID":"d10a100f-6ad4-4ea9-8a30-099f3dd405f8","Type":"ContainerStarted","Data":"4f047fbdba43d6f32d577681501d3e0e8c9db2c0cbe972a85fe67c8e36109b5e"} Jul 24 00:22:00.316628 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:22:00.316596 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" event={"ID":"d10a100f-6ad4-4ea9-8a30-099f3dd405f8","Type":"ContainerStarted","Data":"bbe6c91df553bdd942136e7ae8a2bcf882c3c2030df5ac161ba54f9f73becce1"} Jul 24 00:22:00.317072 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:22:00.316834 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" Jul 24 00:22:00.353703 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:22:00.353649 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" podStartSLOduration=1.878140536 podStartE2EDuration="4.353630494s" totalImagesPullingTime="2.475489958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:22:00.352592169 +0000 UTC m=+496.925688972" watchObservedRunningTime="2026-07-24 00:22:00.353630494 +0000 UTC m=+496.926727296" Jul 24 00:22:11.323066 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:22:11.323034 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-47jp2" Jul 24 00:23:08.293804 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:08.293720 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:23:16.716110 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:16.716075 2576 generic.go:356] "Generic (PLEG): container finished" podID="fa54590a-51b8-4e30-b65c-55089f26bf3c" containerID="3722609907ea14d1f9e42c850f7711c48edf566c7270c887a1e62b4bd078380d" exitCode=0 Jul 24 00:23:16.716527 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:16.716153 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" event={"ID":"fa54590a-51b8-4e30-b65c-55089f26bf3c","Type":"ContainerDied","Data":"3722609907ea14d1f9e42c850f7711c48edf566c7270c887a1e62b4bd078380d"} Jul 24 00:23:16.716578 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:16.716544 2576 scope.go:117] "RemoveContainer" containerID="3722609907ea14d1f9e42c850f7711c48edf566c7270c887a1e62b4bd078380d" Jul 24 00:23:16.717545 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:16.717519 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:23:16.717600 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:16.717584 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:23:16.717635 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:16.717616 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:23:17.722929 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:17.722896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" event={"ID":"fa54590a-51b8-4e30-b65c-55089f26bf3c","Type":"ContainerStarted","Data":"3d002e345b89f65d1fbe00a6e0cf29235a0e20e9573aa8ed57f795cd65941024"} Jul 24 00:23:18.415421 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:18.415386 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:23:18.415619 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:18.415595 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rfkdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 24 00:23:18.415671 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:18.415651 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" podUID="fa54590a-51b8-4e30-b65c-55089f26bf3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 24 00:23:19.416185 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:19.416143 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rfkdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 24 00:23:19.416674 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:19.416211 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" podUID="fa54590a-51b8-4e30-b65c-55089f26bf3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 24 00:23:20.416446 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:20.416393 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rfkdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 24 00:23:20.416832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:20.416463 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" podUID="fa54590a-51b8-4e30-b65c-55089f26bf3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 24 00:23:21.415998 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:21.415931 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rfkdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 24 00:23:21.416180 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:21.416019 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" podUID="fa54590a-51b8-4e30-b65c-55089f26bf3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 24 00:23:22.420066 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:22.420035 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:23:22.420429 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:22.420312 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:23:22.421039 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:22.421025 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rfkdh" Jul 24 00:23:32.489321 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:32.489279 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:23:39.285297 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:39.285257 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:23:43.902737 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:43.902701 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:23:43.903442 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:43.903424 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:23:51.630543 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.630509 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg"] Jul 24 00:23:51.633662 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.633645 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.636494 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.636467 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 00:23:51.637490 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.637470 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 24 00:23:51.637594 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.637488 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 00:23:51.637594 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.637578 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bk7h5\"" Jul 24 00:23:51.647700 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.647678 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg"] Jul 24 00:23:51.758297 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.758263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed33a365-03ea-444d-89e3-38acf8eb2c2c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.758474 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.758302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed33a365-03ea-444d-89e3-38acf8eb2c2c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.758474 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.758330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed33a365-03ea-444d-89e3-38acf8eb2c2c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.758474 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.758440 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed33a365-03ea-444d-89e3-38acf8eb2c2c-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.758630 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.758474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed33a365-03ea-444d-89e3-38acf8eb2c2c-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.758630 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.758521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr56b\" (UniqueName: \"kubernetes.io/projected/ed33a365-03ea-444d-89e3-38acf8eb2c2c-kube-api-access-dr56b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.859457 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.859419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed33a365-03ea-444d-89e3-38acf8eb2c2c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.859640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.859483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed33a365-03ea-444d-89e3-38acf8eb2c2c-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.859640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.859506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed33a365-03ea-444d-89e3-38acf8eb2c2c-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.859640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.859536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr56b\" (UniqueName: \"kubernetes.io/projected/ed33a365-03ea-444d-89e3-38acf8eb2c2c-kube-api-access-dr56b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.859640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.859560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed33a365-03ea-444d-89e3-38acf8eb2c2c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.859640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.859582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed33a365-03ea-444d-89e3-38acf8eb2c2c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.860032 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.860001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed33a365-03ea-444d-89e3-38acf8eb2c2c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.860151 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.860093 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed33a365-03ea-444d-89e3-38acf8eb2c2c-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.860151 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.860120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed33a365-03ea-444d-89e3-38acf8eb2c2c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.861736 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.861709 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed33a365-03ea-444d-89e3-38acf8eb2c2c-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.862009 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.861991 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed33a365-03ea-444d-89e3-38acf8eb2c2c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.867632 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.867610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr56b\" (UniqueName: \"kubernetes.io/projected/ed33a365-03ea-444d-89e3-38acf8eb2c2c-kube-api-access-dr56b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hb4wg\" (UID: \"ed33a365-03ea-444d-89e3-38acf8eb2c2c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:51.943691 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:51.943595 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:23:52.101944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:52.101907 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg"] Jul 24 00:23:52.107859 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:23:52.107828 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded33a365_03ea_444d_89e3_38acf8eb2c2c.slice/crio-1db44c2665feae41a2492ced747ba93deb843a476b06427b3b76a910b1aade95 WatchSource:0}: Error finding container 1db44c2665feae41a2492ced747ba93deb843a476b06427b3b76a910b1aade95: Status 404 returned error can't find the container with id 1db44c2665feae41a2492ced747ba93deb843a476b06427b3b76a910b1aade95 Jul 24 00:23:52.887965 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:52.887918 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:23:52.914010 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:52.913978 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" event={"ID":"ed33a365-03ea-444d-89e3-38acf8eb2c2c","Type":"ContainerStarted","Data":"1db44c2665feae41a2492ced747ba93deb843a476b06427b3b76a910b1aade95"} Jul 24 00:23:53.921657 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:53.921617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" event={"ID":"ed33a365-03ea-444d-89e3-38acf8eb2c2c","Type":"ContainerStarted","Data":"1a1189cb0931463fdcea98aa08dde1528a042fcf72eace11ba75a59fff7b08a9"} Jul 24 00:23:53.938336 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:23:53.938286 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" podStartSLOduration=1.9857586459999998 podStartE2EDuration="2.938239861s" totalImagesPullingTime="952.481215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:23:53.937305504 +0000 UTC m=+610.510402306" watchObservedRunningTime="2026-07-24 00:23:53.938239861 +0000 UTC m=+610.511336664" Jul 24 00:24:00.991574 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:24:00.991537 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:24:01.943728 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:24:01.943691 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:24:01.943906 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:24:01.943754 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:24:01.956032 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:24:01.956005 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:24:01.974136 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:24:01.974105 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hb4wg" Jul 24 00:24:14.991580 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:24:14.991542 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:24:26.288410 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:24:26.288371 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:25:51.386097 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:25:51.386010 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:26:00.792293 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:26:00.792258 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:26:12.392423 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:26:12.392385 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:27:53.190287 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:27:53.190247 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:28:02.485433 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:28:02.485396 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:28:31.583579 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:28:31.583542 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:28:41.398994 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:28:41.398889 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:28:43.940741 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:28:43.940711 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:28:43.941210 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:28:43.940777 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:28:50.989087 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:28:50.989048 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:29:02.287764 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:29:02.287724 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:29:12.690826 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:29:12.690785 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:29:21.887588 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:29:21.887550 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:29:32.791108 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:29:32.791071 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:29:42.383618 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:29:42.383579 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:30:00.139694 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:00.139658 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747550-d9v6f"] Jul 24 00:30:00.143144 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:00.143129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747550-d9v6f" Jul 24 00:30:00.145509 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:00.145492 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\"" Jul 24 00:30:00.152240 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:00.152224 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747550-d9v6f" Jul 24 00:30:00.157466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:00.157442 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747550-d9v6f"] Jul 24 00:30:00.289417 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:00.289383 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747550-d9v6f"] Jul 24 00:30:00.291574 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:30:00.291546 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6569a48f_baf4_422d_a7bd_d966036603fd.slice/crio-b28665068c53b10fecc7b08bab9d5e6a8b24dda4ac69410e2cd47c55ed889778 WatchSource:0}: Error finding container b28665068c53b10fecc7b08bab9d5e6a8b24dda4ac69410e2cd47c55ed889778: Status 404 returned error can't find the container with id b28665068c53b10fecc7b08bab9d5e6a8b24dda4ac69410e2cd47c55ed889778 Jul 24 00:30:00.293310 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:00.293294 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 00:30:00.845718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:00.845681 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747550-d9v6f" event={"ID":"6569a48f-baf4-422d-a7bd-d966036603fd","Type":"ContainerStarted","Data":"b28665068c53b10fecc7b08bab9d5e6a8b24dda4ac69410e2cd47c55ed889778"} Jul 24 00:30:03.862999 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:03.862940 2576 generic.go:356] "Generic (PLEG): container finished" podID="6569a48f-baf4-422d-a7bd-d966036603fd" containerID="1b06d652716ab6a8884515dbaf4cf589cf05bf3b5b659fcf1229e784409ae7fe" exitCode=0 Jul 24 00:30:03.863386 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:03.863010 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747550-d9v6f" event={"ID":"6569a48f-baf4-422d-a7bd-d966036603fd","Type":"ContainerDied","Data":"1b06d652716ab6a8884515dbaf4cf589cf05bf3b5b659fcf1229e784409ae7fe"} Jul 24 00:30:05.003640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:05.003611 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747550-d9v6f" Jul 24 00:30:05.202196 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:05.202113 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:30:05.875805 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:05.875773 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747550-d9v6f" event={"ID":"6569a48f-baf4-422d-a7bd-d966036603fd","Type":"ContainerDied","Data":"b28665068c53b10fecc7b08bab9d5e6a8b24dda4ac69410e2cd47c55ed889778"} Jul 24 00:30:05.875805 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:05.875807 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747550-d9v6f" Jul 24 00:30:05.876084 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:05.875844 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b28665068c53b10fecc7b08bab9d5e6a8b24dda4ac69410e2cd47c55ed889778" Jul 24 00:30:15.688673 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:15.688633 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:30:39.201793 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:39.201752 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:30:49.683348 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:30:49.683308 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:31:05.386316 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:31:05.386275 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:31:14.686042 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:31:14.685996 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:31:26.363399 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:31:26.363355 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:31:35.887960 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:31:35.887920 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:31:45.485092 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:31:45.485004 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:33:35.191956 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:33:35.191905 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:33:43.978221 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:33:43.978191 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:33:43.980764 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:33:43.980744 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:33:46.986813 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:33:46.986775 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:33:55.989065 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:33:55.989026 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:34:07.495593 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:34:07.495551 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:34:19.082543 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:34:19.082505 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:34:29.690977 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:34:29.690930 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:34:38.885694 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:34:38.885614 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:34:49.634279 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:34:49.634245 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:35:00.395674 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:35:00.395638 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:35:09.286910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:35:09.286870 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:35:19.587471 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:35:19.587435 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:35:42.189985 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:35:42.189932 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:35:51.684362 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:35:51.684328 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:36:08.190148 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:08.190065 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:36:14.889339 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:14.889304 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2"] Jul 24 00:36:14.889933 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:14.889758 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6569a48f-baf4-422d-a7bd-d966036603fd" containerName="cleanup" Jul 24 00:36:14.889933 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:14.889778 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6569a48f-baf4-422d-a7bd-d966036603fd" containerName="cleanup" Jul 24 00:36:14.890079 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:14.889935 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6569a48f-baf4-422d-a7bd-d966036603fd" containerName="cleanup" Jul 24 00:36:14.893064 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:14.893044 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:14.895593 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:14.895567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3cc92572-kserve-self-signed-certs\"" Jul 24 00:36:14.904346 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:14.904323 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2"] Jul 24 00:36:15.003827 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.003792 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-model-cache\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.004004 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.003851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztp4t\" (UniqueName: \"kubernetes.io/projected/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-kube-api-access-ztp4t\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.004004 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.003901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-tls-certs\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.004004 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.003932 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-dshm\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.004004 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.003984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-tmp-dir\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.004004 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.004003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-home\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.105119 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.105073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-tls-certs\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.105119 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.105122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-dshm\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.105376 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.105146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-tmp-dir\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.105376 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.105270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-home\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.105485 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.105412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-model-cache\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.105545 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.105480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztp4t\" (UniqueName: \"kubernetes.io/projected/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-kube-api-access-ztp4t\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.105599 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.105557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-tmp-dir\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.105640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.105615 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-home\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.105772 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.105754 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-model-cache\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.107933 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.107901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-dshm\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.108095 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.108076 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-tls-certs\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.112680 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.112656 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztp4t\" (UniqueName: \"kubernetes.io/projected/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-kube-api-access-ztp4t\") pod \"e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.203523 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.203439 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:15.337509 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.337480 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2"] Jul 24 00:36:15.338937 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:36:15.338913 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7ad9d6d_0e7a_411b_acb0_cf88af551a11.slice/crio-54fe8b03518e19bf1bc51bcd74b5572f163723fb9bd0bc82900e735b59d90388 WatchSource:0}: Error finding container 54fe8b03518e19bf1bc51bcd74b5572f163723fb9bd0bc82900e735b59d90388: Status 404 returned error can't find the container with id 54fe8b03518e19bf1bc51bcd74b5572f163723fb9bd0bc82900e735b59d90388 Jul 24 00:36:15.816046 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.816005 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" event={"ID":"a7ad9d6d-0e7a-411b-acb0-cf88af551a11","Type":"ContainerStarted","Data":"b880eadb3ad20ef0348c5fed41ad0cb2688c9c5f6f998de19f03b38119b4e5e3"} Jul 24 00:36:15.816218 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.816056 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" event={"ID":"a7ad9d6d-0e7a-411b-acb0-cf88af551a11","Type":"ContainerStarted","Data":"54fe8b03518e19bf1bc51bcd74b5572f163723fb9bd0bc82900e735b59d90388"} Jul 24 00:36:15.842229 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:15.842180 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" podStartSLOduration=1.842166138 podStartE2EDuration="1.842166138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:36:15.840039628 +0000 UTC m=+1352.413136430" watchObservedRunningTime="2026-07-24 00:36:15.842166138 +0000 UTC m=+1352.415262975" Jul 24 00:36:19.189420 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:19.189383 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:36:25.203928 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:25.203888 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:25.204334 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:25.203976 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:25.216774 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:25.216743 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:25.876520 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:25.876488 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:32.786694 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:32.786662 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2"] Jul 24 00:36:35.190209 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:35.190176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:36:47.805100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:47.805060 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" podUID="a7ad9d6d-0e7a-411b-acb0-cf88af551a11" containerName="main" containerID="cri-o://b880eadb3ad20ef0348c5fed41ad0cb2688c9c5f6f998de19f03b38119b4e5e3" gracePeriod=45 Jul 24 00:36:47.985849 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:47.985823 2576 generic.go:356] "Generic (PLEG): container finished" podID="a7ad9d6d-0e7a-411b-acb0-cf88af551a11" containerID="b880eadb3ad20ef0348c5fed41ad0cb2688c9c5f6f998de19f03b38119b4e5e3" exitCode=0 Jul 24 00:36:47.985997 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:47.985902 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" event={"ID":"a7ad9d6d-0e7a-411b-acb0-cf88af551a11","Type":"ContainerDied","Data":"b880eadb3ad20ef0348c5fed41ad0cb2688c9c5f6f998de19f03b38119b4e5e3"} Jul 24 00:36:48.046201 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.046170 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:48.205356 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.205278 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-home\") pod \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " Jul 24 00:36:48.205356 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.205334 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-model-cache\") pod \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " Jul 24 00:36:48.205356 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.205351 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-dshm\") pod \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " Jul 24 00:36:48.205678 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.205395 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-tls-certs\") pod \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " Jul 24 00:36:48.205678 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.205413 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-tmp-dir\") pod \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " Jul 24 00:36:48.205678 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.205453 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztp4t\" (UniqueName: \"kubernetes.io/projected/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-kube-api-access-ztp4t\") pod \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\" (UID: \"a7ad9d6d-0e7a-411b-acb0-cf88af551a11\") " Jul 24 00:36:48.205678 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.205579 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-home" (OuterVolumeSpecName: "home") pod "a7ad9d6d-0e7a-411b-acb0-cf88af551a11" (UID: "a7ad9d6d-0e7a-411b-acb0-cf88af551a11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:36:48.205678 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.205592 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7ad9d6d-0e7a-411b-acb0-cf88af551a11" (UID: "a7ad9d6d-0e7a-411b-acb0-cf88af551a11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:36:48.205882 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.205682 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7ad9d6d-0e7a-411b-acb0-cf88af551a11" (UID: "a7ad9d6d-0e7a-411b-acb0-cf88af551a11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:36:48.205882 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.205796 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-home\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:36:48.205882 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.205807 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-model-cache\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:36:48.205882 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.205818 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-tmp-dir\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:36:48.207594 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.207571 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7ad9d6d-0e7a-411b-acb0-cf88af551a11" (UID: "a7ad9d6d-0e7a-411b-acb0-cf88af551a11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:36:48.207679 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.207577 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-dshm" (OuterVolumeSpecName: "dshm") pod "a7ad9d6d-0e7a-411b-acb0-cf88af551a11" (UID: "a7ad9d6d-0e7a-411b-acb0-cf88af551a11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:36:48.207679 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.207606 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-kube-api-access-ztp4t" (OuterVolumeSpecName: "kube-api-access-ztp4t") pod "a7ad9d6d-0e7a-411b-acb0-cf88af551a11" (UID: "a7ad9d6d-0e7a-411b-acb0-cf88af551a11"). InnerVolumeSpecName "kube-api-access-ztp4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:36:48.307087 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.307049 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-dshm\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:36:48.307087 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.307081 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-tls-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:36:48.307087 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.307092 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztp4t\" (UniqueName: \"kubernetes.io/projected/a7ad9d6d-0e7a-411b-acb0-cf88af551a11-kube-api-access-ztp4t\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:36:48.790420 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.790384 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:36:48.993075 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.993046 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" Jul 24 00:36:48.993493 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.993086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2" event={"ID":"a7ad9d6d-0e7a-411b-acb0-cf88af551a11","Type":"ContainerDied","Data":"54fe8b03518e19bf1bc51bcd74b5572f163723fb9bd0bc82900e735b59d90388"} Jul 24 00:36:48.993493 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:48.993137 2576 scope.go:117] "RemoveContainer" containerID="b880eadb3ad20ef0348c5fed41ad0cb2688c9c5f6f998de19f03b38119b4e5e3" Jul 24 00:36:49.015807 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:49.015780 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2"] Jul 24 00:36:49.019556 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:49.019533 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3cc92572-kserve-694dbddd47-hr4l2"] Jul 24 00:36:50.000959 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:50.000913 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ad9d6d-0e7a-411b-acb0-cf88af551a11" path="/var/lib/kubelet/pods/a7ad9d6d-0e7a-411b-acb0-cf88af551a11/volumes" Jul 24 00:36:58.086902 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:36:58.086864 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:37:07.396488 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:37:07.396452 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:37:20.783266 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:37:20.783228 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:37:29.694716 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:37:29.694676 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:37:40.305673 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:37:40.305458 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:37:49.286264 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:37:49.286227 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:37:58.496151 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:37:58.496116 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:38:11.753636 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:38:11.753601 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:38:20.691834 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:38:20.691797 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:38:29.989512 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:38:29.989472 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:38:41.913680 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:38:41.913649 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:38:44.017624 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:38:44.017597 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:38:44.021457 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:38:44.021434 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:38:53.393126 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:38:53.393089 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:39:01.992326 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:39:01.992288 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:39:18.980728 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:39:18.980692 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:39:30.441057 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:39:30.441018 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:39:42.485252 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:39:42.485210 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:39:55.696119 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:39:55.696080 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:40:05.881905 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:05.881870 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:40:15.489554 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:15.489514 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:40:16.161282 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.161250 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv"] Jul 24 00:40:16.161594 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.161582 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ad9d6d-0e7a-411b-acb0-cf88af551a11" containerName="main" Jul 24 00:40:16.161636 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.161596 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ad9d6d-0e7a-411b-acb0-cf88af551a11" containerName="main" Jul 24 00:40:16.161670 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.161665 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ad9d6d-0e7a-411b-acb0-cf88af551a11" containerName="main" Jul 24 00:40:16.164711 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.164695 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.167214 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.167186 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-edc8c8a3-openshift-default-dockercfg-nvv2h\"" Jul 24 00:40:16.183734 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.183712 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv"] Jul 24 00:40:16.328290 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.328247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5980af28-36c5-4f2e-9853-70af9a2656f6-istiod-ca-cert\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.328290 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.328293 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-credential-socket\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.328539 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.328347 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-workload-socket\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.328539 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.328368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-podinfo\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.328539 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.328439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-envoy\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.328539 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.328506 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-workload-certs\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.328701 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.328545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88b5s\" (UniqueName: \"kubernetes.io/projected/5980af28-36c5-4f2e-9853-70af9a2656f6-kube-api-access-88b5s\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.328701 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.328574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-data\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.328701 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.328667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-token\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.429439 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.429358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-token\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.429439 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.429399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5980af28-36c5-4f2e-9853-70af9a2656f6-istiod-ca-cert\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.429439 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.429417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-credential-socket\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.429703 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.429448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-workload-socket\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.429703 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.429470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-podinfo\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.429703 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.429499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-envoy\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.429703 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.429548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-workload-certs\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.429703 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.429577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88b5s\" (UniqueName: \"kubernetes.io/projected/5980af28-36c5-4f2e-9853-70af9a2656f6-kube-api-access-88b5s\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.429703 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.429602 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-data\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.430267 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.430180 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5980af28-36c5-4f2e-9853-70af9a2656f6-istiod-ca-cert\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.430411 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.430369 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-workload-socket\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.430463 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.430417 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-workload-certs\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.430463 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.430420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-credential-socket\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.430540 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.430520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-data\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.431989 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.431966 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-envoy\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.432094 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.432036 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-podinfo\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.438097 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.438072 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-token\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.438255 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.438238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88b5s\" (UniqueName: \"kubernetes.io/projected/5980af28-36c5-4f2e-9853-70af9a2656f6-kube-api-access-88b5s\") pod \"e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.475538 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.475511 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:16.610518 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.610489 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv"] Jul 24 00:40:16.612650 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:40:16.612619 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5980af28_36c5_4f2e_9853_70af9a2656f6.slice/crio-01a82ac27d6d4268f686709b7a48951d5650e68a9309e06e8ed04c5c9b2b5c9e WatchSource:0}: Error finding container 01a82ac27d6d4268f686709b7a48951d5650e68a9309e06e8ed04c5c9b2b5c9e: Status 404 returned error can't find the container with id 01a82ac27d6d4268f686709b7a48951d5650e68a9309e06e8ed04c5c9b2b5c9e Jul 24 00:40:16.615150 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.615122 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:40:16.615235 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.615181 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:40:16.615235 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:16.615218 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:40:17.056525 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:17.056486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" event={"ID":"5980af28-36c5-4f2e-9853-70af9a2656f6","Type":"ContainerStarted","Data":"ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15"} Jul 24 00:40:17.056734 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:17.056550 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" event={"ID":"5980af28-36c5-4f2e-9853-70af9a2656f6","Type":"ContainerStarted","Data":"01a82ac27d6d4268f686709b7a48951d5650e68a9309e06e8ed04c5c9b2b5c9e"} Jul 24 00:40:17.107049 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:17.106997 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" podStartSLOduration=1.106979439 podStartE2EDuration="1.106979439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:40:17.105729128 +0000 UTC m=+1593.678825942" watchObservedRunningTime="2026-07-24 00:40:17.106979439 +0000 UTC m=+1593.680076241" Jul 24 00:40:17.476395 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:17.476315 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:17.480874 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:17.480848 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:18.061572 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:18.061546 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:18.062627 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:18.062611 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:26.395825 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:26.395782 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:40:34.103117 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:34.103077 2576 patch_prober.go:28] interesting pod/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 00:40:34.105352 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:34.103150 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 00:40:35.159158 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:35.159120 2576 generic.go:356] "Generic (PLEG): container finished" podID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerID="ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15" exitCode=0 Jul 24 00:40:35.159542 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:35.159189 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" event={"ID":"5980af28-36c5-4f2e-9853-70af9a2656f6","Type":"ContainerDied","Data":"ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15"} Jul 24 00:40:35.159609 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:35.159593 2576 scope.go:117] "RemoveContainer" containerID="ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15" Jul 24 00:40:35.160594 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:35.160568 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:40:35.160648 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:35.160633 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:40:35.160685 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:35.160669 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:40:36.166608 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:36.166577 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" event={"ID":"5980af28-36c5-4f2e-9853-70af9a2656f6","Type":"ContainerStarted","Data":"2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3"} Jul 24 00:40:36.476597 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:36.476463 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:36.476773 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:36.476748 2576 patch_prober.go:28] interesting pod/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 24 00:40:36.476822 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:36.476800 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 24 00:40:37.476856 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:37.476824 2576 patch_prober.go:28] interesting pod/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 24 00:40:37.477257 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:37.476876 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 24 00:40:38.476236 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:38.476194 2576 patch_prober.go:28] interesting pod/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 24 00:40:38.476419 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:38.476260 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 24 00:40:39.476760 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:39.476724 2576 patch_prober.go:28] interesting pod/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 24 00:40:39.477170 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:39.476781 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 24 00:40:40.479720 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:40.479691 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:40.480077 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:40.480038 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:40.480756 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:40.480733 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:40:44.111528 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:44.111499 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh"] Jul 24 00:40:44.114994 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:44.114977 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh" Jul 24 00:40:44.117155 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:44.117135 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 00:40:44.117283 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:44.117217 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\"" Jul 24 00:40:44.122429 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:44.122017 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh"] Jul 24 00:40:44.190325 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:44.190290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c168d5d0-3d8f-470a-a262-74ffe6918127-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh\" (UID: \"c168d5d0-3d8f-470a-a262-74ffe6918127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh" Jul 24 00:40:44.291088 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:44.291059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c168d5d0-3d8f-470a-a262-74ffe6918127-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh\" (UID: \"c168d5d0-3d8f-470a-a262-74ffe6918127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh" Jul 24 00:40:44.291717 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:44.291697 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c168d5d0-3d8f-470a-a262-74ffe6918127-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh\" (UID: \"c168d5d0-3d8f-470a-a262-74ffe6918127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh" Jul 24 00:40:44.426686 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:44.426606 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh" Jul 24 00:40:44.556065 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:44.556037 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh"] Jul 24 00:40:44.558652 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:40:44.558626 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc168d5d0_3d8f_470a_a262_74ffe6918127.slice/crio-d5b39c2d4d001691047ce2f8d138abd584deb4eab298742bd40ffbaed820b668 WatchSource:0}: Error finding container d5b39c2d4d001691047ce2f8d138abd584deb4eab298742bd40ffbaed820b668: Status 404 returned error can't find the container with id d5b39c2d4d001691047ce2f8d138abd584deb4eab298742bd40ffbaed820b668 Jul 24 00:40:45.215106 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:45.215071 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh" event={"ID":"c168d5d0-3d8f-470a-a262-74ffe6918127","Type":"ContainerStarted","Data":"9fd2d4a7bb8947d719de260e1433b542549357a04efe0c8dcbb9d3e2017977a8"} Jul 24 00:40:45.215504 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:45.215122 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh" event={"ID":"c168d5d0-3d8f-470a-a262-74ffe6918127","Type":"ContainerStarted","Data":"d5b39c2d4d001691047ce2f8d138abd584deb4eab298742bd40ffbaed820b668"} Jul 24 00:40:45.229612 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:45.229563 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh" podStartSLOduration=1.229523026 podStartE2EDuration="1.229523026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:40:45.228640448 +0000 UTC m=+1621.801737263" watchObservedRunningTime="2026-07-24 00:40:45.229523026 +0000 UTC m=+1621.802619826" Jul 24 00:40:46.222710 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:46.222675 2576 generic.go:356] "Generic (PLEG): container finished" podID="c168d5d0-3d8f-470a-a262-74ffe6918127" containerID="9fd2d4a7bb8947d719de260e1433b542549357a04efe0c8dcbb9d3e2017977a8" exitCode=0 Jul 24 00:40:46.223120 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:46.222759 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh" event={"ID":"c168d5d0-3d8f-470a-a262-74ffe6918127","Type":"ContainerDied","Data":"9fd2d4a7bb8947d719de260e1433b542549357a04efe0c8dcbb9d3e2017977a8"} Jul 24 00:40:47.359296 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:47.359274 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh" Jul 24 00:40:47.417342 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:47.417311 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c168d5d0-3d8f-470a-a262-74ffe6918127-maas-api-service-ca\") pod \"c168d5d0-3d8f-470a-a262-74ffe6918127\" (UID: \"c168d5d0-3d8f-470a-a262-74ffe6918127\") " Jul 24 00:40:47.417701 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:47.417678 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c168d5d0-3d8f-470a-a262-74ffe6918127-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c168d5d0-3d8f-470a-a262-74ffe6918127" (UID: "c168d5d0-3d8f-470a-a262-74ffe6918127"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:40:47.518973 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:47.518879 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c168d5d0-3d8f-470a-a262-74ffe6918127-maas-api-service-ca\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:40:48.236763 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:48.236733 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh" Jul 24 00:40:48.236763 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:48.236744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh" event={"ID":"c168d5d0-3d8f-470a-a262-74ffe6918127","Type":"ContainerDied","Data":"d5b39c2d4d001691047ce2f8d138abd584deb4eab298742bd40ffbaed820b668"} Jul 24 00:40:48.236975 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:48.236788 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5b39c2d4d001691047ce2f8d138abd584deb4eab298742bd40ffbaed820b668" Jul 24 00:40:57.185737 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:57.185696 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:40:59.184968 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:59.184924 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh"] Jul 24 00:40:59.187126 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:59.187098 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-edc8c8a3-45vgh"] Jul 24 00:40:59.329493 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:59.329456 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv"] Jul 24 00:40:59.329699 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:40:59.329673 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerName="istio-proxy" containerID="cri-o://2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3" gracePeriod=30 Jul 24 00:41:00.001521 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.001483 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c168d5d0-3d8f-470a-a262-74ffe6918127" path="/var/lib/kubelet/pods/c168d5d0-3d8f-470a-a262-74ffe6918127/volumes" Jul 24 00:41:00.404917 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.404887 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj"] Jul 24 00:41:00.405354 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.405260 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c168d5d0-3d8f-470a-a262-74ffe6918127" containerName="revoke-keys" Jul 24 00:41:00.405354 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.405274 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c168d5d0-3d8f-470a-a262-74ffe6918127" containerName="revoke-keys" Jul 24 00:41:00.405354 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.405336 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c168d5d0-3d8f-470a-a262-74ffe6918127" containerName="revoke-keys" Jul 24 00:41:00.408592 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.408576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.410767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.410749 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-eaba160e-gw-openshift-default-dockercfg-vmtxt\"" Jul 24 00:41:00.422097 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.422074 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj"] Jul 24 00:41:00.541018 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.540984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-istio-data\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.541198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.541023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8hwp\" (UniqueName: \"kubernetes.io/projected/a2154268-aa02-4a08-971e-62f73304f17f-kube-api-access-p8hwp\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.541198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.541050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2154268-aa02-4a08-971e-62f73304f17f-istio-podinfo\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.541198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.541147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-workload-socket\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.541198 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.541182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-istio-envoy\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.541357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.541202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2154268-aa02-4a08-971e-62f73304f17f-istio-token\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.541357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.541225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-credential-socket\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.541357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.541251 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-workload-certs\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.541357 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.541300 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2154268-aa02-4a08-971e-62f73304f17f-istiod-ca-cert\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.642031 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.641995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-istio-data\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.642031 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.642034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hwp\" (UniqueName: \"kubernetes.io/projected/a2154268-aa02-4a08-971e-62f73304f17f-kube-api-access-p8hwp\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.642262 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.642152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2154268-aa02-4a08-971e-62f73304f17f-istio-podinfo\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.642262 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.642218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-workload-socket\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.642262 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.642255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-istio-envoy\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.642435 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.642282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2154268-aa02-4a08-971e-62f73304f17f-istio-token\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.642435 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.642322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-credential-socket\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.642435 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.642349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-workload-certs\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.642435 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.642412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2154268-aa02-4a08-971e-62f73304f17f-istiod-ca-cert\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.642666 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.642434 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-istio-data\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.642666 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.642575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-workload-socket\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.642781 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.642758 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-workload-certs\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.644652 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.643166 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-credential-socket\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.644652 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.643313 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2154268-aa02-4a08-971e-62f73304f17f-istiod-ca-cert\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.644652 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.644549 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-istio-envoy\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.650922 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.650883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2154268-aa02-4a08-971e-62f73304f17f-istio-podinfo\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.652907 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.652881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2154268-aa02-4a08-971e-62f73304f17f-istio-token\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.653176 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.653160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hwp\" (UniqueName: \"kubernetes.io/projected/a2154268-aa02-4a08-971e-62f73304f17f-kube-api-access-p8hwp\") pod \"e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.719343 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.719256 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:00.858896 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.858868 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj"] Jul 24 00:41:00.860446 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:41:00.860419 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2154268_aa02_4a08_971e_62f73304f17f.slice/crio-9dce2083fdab9dfe9ce2a5207d16ce0fb1adbe0c09ce08747bdbf177da59b55f WatchSource:0}: Error finding container 9dce2083fdab9dfe9ce2a5207d16ce0fb1adbe0c09ce08747bdbf177da59b55f: Status 404 returned error can't find the container with id 9dce2083fdab9dfe9ce2a5207d16ce0fb1adbe0c09ce08747bdbf177da59b55f Jul 24 00:41:00.862622 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.862591 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:41:00.862717 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.862665 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:41:00.862717 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:00.862705 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:41:01.307023 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:01.306983 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" event={"ID":"a2154268-aa02-4a08-971e-62f73304f17f","Type":"ContainerStarted","Data":"c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc"} Jul 24 00:41:01.307219 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:01.307055 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" event={"ID":"a2154268-aa02-4a08-971e-62f73304f17f","Type":"ContainerStarted","Data":"9dce2083fdab9dfe9ce2a5207d16ce0fb1adbe0c09ce08747bdbf177da59b55f"} Jul 24 00:41:01.329229 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:01.329178 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" podStartSLOduration=1.329161324 podStartE2EDuration="1.329161324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:41:01.327544644 +0000 UTC m=+1637.900641454" watchObservedRunningTime="2026-07-24 00:41:01.329161324 +0000 UTC m=+1637.902258125" Jul 24 00:41:01.720315 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:01.720223 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:01.725124 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:01.725098 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:02.312531 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:02.312502 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:02.313293 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:02.313272 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:04.588773 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.588711 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:41:04.679664 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.679629 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-workload-socket\") pod \"5980af28-36c5-4f2e-9853-70af9a2656f6\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " Jul 24 00:41:04.679860 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.679692 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5980af28-36c5-4f2e-9853-70af9a2656f6-istiod-ca-cert\") pod \"5980af28-36c5-4f2e-9853-70af9a2656f6\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " Jul 24 00:41:04.679860 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.679736 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-credential-socket\") pod \"5980af28-36c5-4f2e-9853-70af9a2656f6\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " Jul 24 00:41:04.679860 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.679757 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-podinfo\") pod \"5980af28-36c5-4f2e-9853-70af9a2656f6\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " Jul 24 00:41:04.679860 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.679779 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-envoy\") pod \"5980af28-36c5-4f2e-9853-70af9a2656f6\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " Jul 24 00:41:04.679860 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.679860 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-data\") pod \"5980af28-36c5-4f2e-9853-70af9a2656f6\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " Jul 24 00:41:04.680168 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.679883 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-token\") pod \"5980af28-36c5-4f2e-9853-70af9a2656f6\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " Jul 24 00:41:04.680168 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.679909 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88b5s\" (UniqueName: \"kubernetes.io/projected/5980af28-36c5-4f2e-9853-70af9a2656f6-kube-api-access-88b5s\") pod \"5980af28-36c5-4f2e-9853-70af9a2656f6\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " Jul 24 00:41:04.680168 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.679941 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-workload-certs\") pod \"5980af28-36c5-4f2e-9853-70af9a2656f6\" (UID: \"5980af28-36c5-4f2e-9853-70af9a2656f6\") " Jul 24 00:41:04.680168 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.680024 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5980af28-36c5-4f2e-9853-70af9a2656f6" (UID: "5980af28-36c5-4f2e-9853-70af9a2656f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:41:04.680168 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.680092 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5980af28-36c5-4f2e-9853-70af9a2656f6" (UID: "5980af28-36c5-4f2e-9853-70af9a2656f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:41:04.680422 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.680291 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5980af28-36c5-4f2e-9853-70af9a2656f6" (UID: "5980af28-36c5-4f2e-9853-70af9a2656f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:41:04.680422 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.680369 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:04.680422 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.680391 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:04.680765 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.680742 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5980af28-36c5-4f2e-9853-70af9a2656f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5980af28-36c5-4f2e-9853-70af9a2656f6" (UID: "5980af28-36c5-4f2e-9853-70af9a2656f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:41:04.680900 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.680873 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "5980af28-36c5-4f2e-9853-70af9a2656f6" (UID: "5980af28-36c5-4f2e-9853-70af9a2656f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:41:04.682270 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.682238 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "5980af28-36c5-4f2e-9853-70af9a2656f6" (UID: "5980af28-36c5-4f2e-9853-70af9a2656f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:41:04.682270 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.682255 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5980af28-36c5-4f2e-9853-70af9a2656f6" (UID: "5980af28-36c5-4f2e-9853-70af9a2656f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:41:04.682410 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.682283 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5980af28-36c5-4f2e-9853-70af9a2656f6" (UID: "5980af28-36c5-4f2e-9853-70af9a2656f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 00:41:04.682410 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.682297 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5980af28-36c5-4f2e-9853-70af9a2656f6-kube-api-access-88b5s" (OuterVolumeSpecName: "kube-api-access-88b5s") pod "5980af28-36c5-4f2e-9853-70af9a2656f6" (UID: "5980af28-36c5-4f2e-9853-70af9a2656f6"). InnerVolumeSpecName "kube-api-access-88b5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:41:04.781640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.781613 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:04.781640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.781636 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:04.781640 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.781645 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88b5s\" (UniqueName: \"kubernetes.io/projected/5980af28-36c5-4f2e-9853-70af9a2656f6-kube-api-access-88b5s\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:04.781840 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.781653 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:04.781840 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.781664 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5980af28-36c5-4f2e-9853-70af9a2656f6-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:04.781840 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.781671 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:04.781840 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:04.781677 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5980af28-36c5-4f2e-9853-70af9a2656f6-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:05.330320 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:05.330288 2576 generic.go:356] "Generic (PLEG): container finished" podID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerID="2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3" exitCode=0 Jul 24 00:41:05.330516 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:05.330358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" event={"ID":"5980af28-36c5-4f2e-9853-70af9a2656f6","Type":"ContainerDied","Data":"2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3"} Jul 24 00:41:05.330516 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:05.330371 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" Jul 24 00:41:05.330516 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:05.330399 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv" event={"ID":"5980af28-36c5-4f2e-9853-70af9a2656f6","Type":"ContainerDied","Data":"01a82ac27d6d4268f686709b7a48951d5650e68a9309e06e8ed04c5c9b2b5c9e"} Jul 24 00:41:05.330516 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:05.330422 2576 scope.go:117] "RemoveContainer" containerID="2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3" Jul 24 00:41:05.343471 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:05.341346 2576 scope.go:117] "RemoveContainer" containerID="ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15" Jul 24 00:41:05.352741 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:05.352721 2576 scope.go:117] "RemoveContainer" containerID="2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3" Jul 24 00:41:05.353048 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:41:05.353021 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3\": container with ID starting with 2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3 not found: ID does not exist" containerID="2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3" Jul 24 00:41:05.353164 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:05.353051 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3"} err="failed to get container status \"2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3\": rpc error: code = NotFound desc = could not find container \"2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3\": container with ID starting with 2ad4b662acc429f1c480d9c07f28dbd260e6fddd51bc7d048722e5ae923272d3 not found: ID does not exist" Jul 24 00:41:05.353164 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:05.353069 2576 scope.go:117] "RemoveContainer" containerID="ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15" Jul 24 00:41:05.353326 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:41:05.353312 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15\": container with ID starting with ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15 not found: ID does not exist" containerID="ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15" Jul 24 00:41:05.353372 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:05.353328 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15"} err="failed to get container status \"ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15\": rpc error: code = NotFound desc = could not find container \"ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15\": container with ID starting with ac506f9b7c8fbe3a6018242bab54fe81ca3310d1f10ea0d1e31422e80545dd15 not found: ID does not exist" Jul 24 00:41:05.359887 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:05.359863 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv"] Jul 24 00:41:05.363497 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:05.363473 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-edc8c8a3-openshift-default-85ffff67c8-xh9fv"] Jul 24 00:41:06.002376 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:06.002341 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" path="/var/lib/kubelet/pods/5980af28-36c5-4f2e-9853-70af9a2656f6/volumes" Jul 24 00:41:07.987413 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:07.987373 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:41:16.400023 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:16.399981 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2154268-aa02-4a08-971e-62f73304f17f" containerID="c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc" exitCode=0 Jul 24 00:41:16.400518 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:16.400036 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" event={"ID":"a2154268-aa02-4a08-971e-62f73304f17f","Type":"ContainerDied","Data":"c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc"} Jul 24 00:41:16.400518 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:16.400413 2576 scope.go:117] "RemoveContainer" containerID="c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc" Jul 24 00:41:16.401390 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:16.401366 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:41:16.401455 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:16.401442 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:41:16.401508 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:16.401486 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:41:17.312465 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:17.312433 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:17.407590 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:17.407552 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" event={"ID":"a2154268-aa02-4a08-971e-62f73304f17f","Type":"ContainerStarted","Data":"101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750"} Jul 24 00:41:17.719497 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:17.719404 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:17.720780 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:17.720754 2576 patch_prober.go:28] interesting pod/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 24 00:41:17.720899 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:17.720813 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" podUID="a2154268-aa02-4a08-971e-62f73304f17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 24 00:41:18.720283 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:18.720242 2576 patch_prober.go:28] interesting pod/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 24 00:41:18.720684 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:18.720313 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" podUID="a2154268-aa02-4a08-971e-62f73304f17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 24 00:41:19.720229 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:19.720190 2576 patch_prober.go:28] interesting pod/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 24 00:41:19.720420 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:19.720261 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" podUID="a2154268-aa02-4a08-971e-62f73304f17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 24 00:41:20.720115 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:20.720078 2576 patch_prober.go:28] interesting pod/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 24 00:41:20.720297 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:20.720150 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" podUID="a2154268-aa02-4a08-971e-62f73304f17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 24 00:41:21.724207 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:21.724177 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:21.724565 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:21.724441 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:21.725191 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:21.725164 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:27.850117 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.850073 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw"] Jul 24 00:41:27.850582 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.850438 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerName="istio-proxy" Jul 24 00:41:27.850582 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.850449 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerName="istio-proxy" Jul 24 00:41:27.850582 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.850459 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerName="istio-proxy" Jul 24 00:41:27.850582 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.850464 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerName="istio-proxy" Jul 24 00:41:27.850582 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.850522 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerName="istio-proxy" Jul 24 00:41:27.850582 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.850530 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5980af28-36c5-4f2e-9853-70af9a2656f6" containerName="istio-proxy" Jul 24 00:41:27.852510 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.852485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw" Jul 24 00:41:27.854858 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.854830 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 00:41:27.854993 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.854869 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\"" Jul 24 00:41:27.860613 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.860587 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw"] Jul 24 00:41:27.893028 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.892987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01df7321-b74a-4e37-87f5-bd1e44c458d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw\" (UID: \"01df7321-b74a-4e37-87f5-bd1e44c458d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw" Jul 24 00:41:27.994051 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.994011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01df7321-b74a-4e37-87f5-bd1e44c458d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw\" (UID: \"01df7321-b74a-4e37-87f5-bd1e44c458d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw" Jul 24 00:41:27.994643 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:27.994622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01df7321-b74a-4e37-87f5-bd1e44c458d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw\" (UID: \"01df7321-b74a-4e37-87f5-bd1e44c458d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw" Jul 24 00:41:28.166367 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:28.166275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw" Jul 24 00:41:28.299582 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:28.299550 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw"] Jul 24 00:41:28.300512 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:41:28.300486 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01df7321_b74a_4e37_87f5_bd1e44c458d1.slice/crio-b98b61598b1c707a1b06025dda56ed5481f261dd49ad985301d23c1aab070c78 WatchSource:0}: Error finding container b98b61598b1c707a1b06025dda56ed5481f261dd49ad985301d23c1aab070c78: Status 404 returned error can't find the container with id b98b61598b1c707a1b06025dda56ed5481f261dd49ad985301d23c1aab070c78 Jul 24 00:41:28.479217 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:28.479189 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw" event={"ID":"01df7321-b74a-4e37-87f5-bd1e44c458d1","Type":"ContainerStarted","Data":"4cd766af51596908b090df937a638228cda7d45927d7b5713e6a03814755ac34"} Jul 24 00:41:28.479346 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:28.479276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw" event={"ID":"01df7321-b74a-4e37-87f5-bd1e44c458d1","Type":"ContainerStarted","Data":"b98b61598b1c707a1b06025dda56ed5481f261dd49ad985301d23c1aab070c78"} Jul 24 00:41:28.494170 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:28.494126 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw" podStartSLOduration=1.4941118709999999 podStartE2EDuration="1.494111871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:41:28.493055628 +0000 UTC m=+1665.066152433" watchObservedRunningTime="2026-07-24 00:41:28.494111871 +0000 UTC m=+1665.067208672" Jul 24 00:41:29.486066 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:29.486032 2576 generic.go:356] "Generic (PLEG): container finished" podID="01df7321-b74a-4e37-87f5-bd1e44c458d1" containerID="4cd766af51596908b090df937a638228cda7d45927d7b5713e6a03814755ac34" exitCode=0 Jul 24 00:41:29.486452 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:29.486081 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw" event={"ID":"01df7321-b74a-4e37-87f5-bd1e44c458d1","Type":"ContainerDied","Data":"4cd766af51596908b090df937a638228cda7d45927d7b5713e6a03814755ac34"} Jul 24 00:41:30.621335 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:30.621311 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw" Jul 24 00:41:30.718032 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:30.718000 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01df7321-b74a-4e37-87f5-bd1e44c458d1-maas-api-service-ca\") pod \"01df7321-b74a-4e37-87f5-bd1e44c458d1\" (UID: \"01df7321-b74a-4e37-87f5-bd1e44c458d1\") " Jul 24 00:41:30.718328 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:30.718306 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01df7321-b74a-4e37-87f5-bd1e44c458d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01df7321-b74a-4e37-87f5-bd1e44c458d1" (UID: "01df7321-b74a-4e37-87f5-bd1e44c458d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:41:30.818685 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:30.818657 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01df7321-b74a-4e37-87f5-bd1e44c458d1-maas-api-service-ca\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:31.497989 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:31.497939 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw" Jul 24 00:41:31.497989 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:31.497972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw" event={"ID":"01df7321-b74a-4e37-87f5-bd1e44c458d1","Type":"ContainerDied","Data":"b98b61598b1c707a1b06025dda56ed5481f261dd49ad985301d23c1aab070c78"} Jul 24 00:41:31.498204 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:31.498016 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b98b61598b1c707a1b06025dda56ed5481f261dd49ad985301d23c1aab070c78" Jul 24 00:41:41.290358 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:41.290317 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:41:42.996538 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:42.996483 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw"] Jul 24 00:41:43.001366 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:43.001334 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eaba160e-x92vw"] Jul 24 00:41:43.124044 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:43.124006 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj"] Jul 24 00:41:43.124258 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:43.124228 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" podUID="a2154268-aa02-4a08-971e-62f73304f17f" containerName="istio-proxy" containerID="cri-o://101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750" gracePeriod=30 Jul 24 00:41:43.969636 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:43.969595 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn"] Jul 24 00:41:43.969997 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:43.969980 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01df7321-b74a-4e37-87f5-bd1e44c458d1" containerName="revoke-keys" Jul 24 00:41:43.970085 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:43.970002 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01df7321-b74a-4e37-87f5-bd1e44c458d1" containerName="revoke-keys" Jul 24 00:41:43.970138 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:43.970120 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01df7321-b74a-4e37-87f5-bd1e44c458d1" containerName="revoke-keys" Jul 24 00:41:43.972094 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:43.972078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:43.974621 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:43.974599 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a227ad09-openshift-default-dockercfg-mqg6b\"" Jul 24 00:41:43.984924 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:43.984897 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn"] Jul 24 00:41:44.002422 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.002394 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01df7321-b74a-4e37-87f5-bd1e44c458d1" path="/var/lib/kubelet/pods/01df7321-b74a-4e37-87f5-bd1e44c458d1/volumes" Jul 24 00:41:44.041514 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.041469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-workload-certs\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.041735 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.041531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-envoy\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.041735 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.041569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istiod-ca-cert\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.041735 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.041594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-data\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.041735 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.041654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-credential-socket\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.041976 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.041765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-workload-socket\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.041976 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.041792 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-podinfo\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.041976 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.041816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-token\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.041976 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.041844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvpgf\" (UniqueName: \"kubernetes.io/projected/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-kube-api-access-cvpgf\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.142646 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.142613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-workload-certs\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.142833 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.142652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-envoy\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.142833 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.142674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istiod-ca-cert\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.142833 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.142692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-data\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.142833 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.142719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-credential-socket\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.142833 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.142793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-workload-socket\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.143083 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.142926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-podinfo\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.143083 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.142993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-token\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.143083 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.143025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvpgf\" (UniqueName: \"kubernetes.io/projected/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-kube-api-access-cvpgf\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.143236 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.143160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-workload-certs\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.143528 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.143508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-credential-socket\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.143613 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.143527 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istiod-ca-cert\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.143613 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.143567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-data\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.143613 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.143597 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-workload-socket\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.144995 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.144970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-envoy\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.145310 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.145290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-podinfo\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.150654 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.150623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-token\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.150818 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.150799 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvpgf\" (UniqueName: \"kubernetes.io/projected/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-kube-api-access-cvpgf\") pod \"e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.282266 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.282171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:44.429674 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.429649 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn"] Jul 24 00:41:44.432035 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:41:44.432007 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f3d507_4a11_4e04_bf9c_ecf4b1d4e8c6.slice/crio-24f9e6a815012b0cfad088ee32ff86200ba87855ecc03db09e42f3ad06223868 WatchSource:0}: Error finding container 24f9e6a815012b0cfad088ee32ff86200ba87855ecc03db09e42f3ad06223868: Status 404 returned error can't find the container with id 24f9e6a815012b0cfad088ee32ff86200ba87855ecc03db09e42f3ad06223868 Jul 24 00:41:44.434294 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.434256 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:41:44.434420 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.434347 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:41:44.434420 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.434394 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:41:44.572535 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.572488 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" event={"ID":"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6","Type":"ContainerStarted","Data":"51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704"} Jul 24 00:41:44.572737 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.572558 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" event={"ID":"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6","Type":"ContainerStarted","Data":"24f9e6a815012b0cfad088ee32ff86200ba87855ecc03db09e42f3ad06223868"} Jul 24 00:41:44.593287 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:44.593233 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" podStartSLOduration=1.593215942 podStartE2EDuration="1.593215942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:41:44.590899587 +0000 UTC m=+1681.163996388" watchObservedRunningTime="2026-07-24 00:41:44.593215942 +0000 UTC m=+1681.166312743" Jul 24 00:41:45.282786 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:45.282746 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:45.287828 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:45.287803 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:45.577648 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:45.577617 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:45.578785 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:45.578765 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:41:48.387216 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.387155 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:48.484024 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.483985 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-istio-data\") pod \"a2154268-aa02-4a08-971e-62f73304f17f\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " Jul 24 00:41:48.484216 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484055 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2154268-aa02-4a08-971e-62f73304f17f-istio-token\") pod \"a2154268-aa02-4a08-971e-62f73304f17f\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " Jul 24 00:41:48.484216 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484080 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2154268-aa02-4a08-971e-62f73304f17f-istio-podinfo\") pod \"a2154268-aa02-4a08-971e-62f73304f17f\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " Jul 24 00:41:48.484216 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484144 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-istio-envoy\") pod \"a2154268-aa02-4a08-971e-62f73304f17f\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " Jul 24 00:41:48.484216 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484185 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-credential-socket\") pod \"a2154268-aa02-4a08-971e-62f73304f17f\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " Jul 24 00:41:48.484216 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484207 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-workload-certs\") pod \"a2154268-aa02-4a08-971e-62f73304f17f\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " Jul 24 00:41:48.484472 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484254 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2154268-aa02-4a08-971e-62f73304f17f-istiod-ca-cert\") pod \"a2154268-aa02-4a08-971e-62f73304f17f\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " Jul 24 00:41:48.484472 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484323 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-workload-socket\") pod \"a2154268-aa02-4a08-971e-62f73304f17f\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " Jul 24 00:41:48.484472 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484352 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8hwp\" (UniqueName: \"kubernetes.io/projected/a2154268-aa02-4a08-971e-62f73304f17f-kube-api-access-p8hwp\") pod \"a2154268-aa02-4a08-971e-62f73304f17f\" (UID: \"a2154268-aa02-4a08-971e-62f73304f17f\") " Jul 24 00:41:48.484622 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484544 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a2154268-aa02-4a08-971e-62f73304f17f" (UID: "a2154268-aa02-4a08-971e-62f73304f17f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:41:48.484622 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484567 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a2154268-aa02-4a08-971e-62f73304f17f" (UID: "a2154268-aa02-4a08-971e-62f73304f17f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:41:48.484730 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484709 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:48.484784 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484728 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:48.484840 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484781 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2154268-aa02-4a08-971e-62f73304f17f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a2154268-aa02-4a08-971e-62f73304f17f" (UID: "a2154268-aa02-4a08-971e-62f73304f17f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:41:48.484840 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484797 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a2154268-aa02-4a08-971e-62f73304f17f" (UID: "a2154268-aa02-4a08-971e-62f73304f17f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:41:48.484942 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.484885 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-istio-data" (OuterVolumeSpecName: "istio-data") pod "a2154268-aa02-4a08-971e-62f73304f17f" (UID: "a2154268-aa02-4a08-971e-62f73304f17f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:41:48.486399 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.486369 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2154268-aa02-4a08-971e-62f73304f17f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a2154268-aa02-4a08-971e-62f73304f17f" (UID: "a2154268-aa02-4a08-971e-62f73304f17f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 00:41:48.486521 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.486444 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a2154268-aa02-4a08-971e-62f73304f17f" (UID: "a2154268-aa02-4a08-971e-62f73304f17f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:41:48.486521 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.486466 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2154268-aa02-4a08-971e-62f73304f17f-istio-token" (OuterVolumeSpecName: "istio-token") pod "a2154268-aa02-4a08-971e-62f73304f17f" (UID: "a2154268-aa02-4a08-971e-62f73304f17f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:41:48.486598 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.486539 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2154268-aa02-4a08-971e-62f73304f17f-kube-api-access-p8hwp" (OuterVolumeSpecName: "kube-api-access-p8hwp") pod "a2154268-aa02-4a08-971e-62f73304f17f" (UID: "a2154268-aa02-4a08-971e-62f73304f17f"). InnerVolumeSpecName "kube-api-access-p8hwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:41:48.585507 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.585474 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2154268-aa02-4a08-971e-62f73304f17f-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:48.585507 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.585500 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:48.585507 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.585509 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8hwp\" (UniqueName: \"kubernetes.io/projected/a2154268-aa02-4a08-971e-62f73304f17f-kube-api-access-p8hwp\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:48.585735 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.585519 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:48.585735 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.585530 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2154268-aa02-4a08-971e-62f73304f17f-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:48.585735 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.585538 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2154268-aa02-4a08-971e-62f73304f17f-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:48.585735 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.585545 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2154268-aa02-4a08-971e-62f73304f17f-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:41:48.597277 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.597248 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2154268-aa02-4a08-971e-62f73304f17f" containerID="101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750" exitCode=0 Jul 24 00:41:48.597421 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.597288 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" Jul 24 00:41:48.597421 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.597317 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" event={"ID":"a2154268-aa02-4a08-971e-62f73304f17f","Type":"ContainerDied","Data":"101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750"} Jul 24 00:41:48.597421 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.597382 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj" event={"ID":"a2154268-aa02-4a08-971e-62f73304f17f","Type":"ContainerDied","Data":"9dce2083fdab9dfe9ce2a5207d16ce0fb1adbe0c09ce08747bdbf177da59b55f"} Jul 24 00:41:48.597421 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.597403 2576 scope.go:117] "RemoveContainer" containerID="101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750" Jul 24 00:41:48.607488 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.607462 2576 scope.go:117] "RemoveContainer" containerID="c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc" Jul 24 00:41:48.619457 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.619434 2576 scope.go:117] "RemoveContainer" containerID="101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750" Jul 24 00:41:48.619751 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:41:48.619731 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750\": container with ID starting with 101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750 not found: ID does not exist" containerID="101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750" Jul 24 00:41:48.619806 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.619761 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750"} err="failed to get container status \"101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750\": rpc error: code = NotFound desc = could not find container \"101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750\": container with ID starting with 101448758c6a49e76e2309cc1367fc69847bdaa67d7435f9c90c2b98e75ce750 not found: ID does not exist" Jul 24 00:41:48.619806 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.619779 2576 scope.go:117] "RemoveContainer" containerID="c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc" Jul 24 00:41:48.620074 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:41:48.620054 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc\": container with ID starting with c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc not found: ID does not exist" containerID="c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc" Jul 24 00:41:48.620140 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.620080 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc"} err="failed to get container status \"c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc\": rpc error: code = NotFound desc = could not find container \"c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc\": container with ID starting with c59670af8010012f9f28e0a0add26e8224f3553d9521d65fc95f0dbe996f93cc not found: ID does not exist" Jul 24 00:41:48.624006 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.623981 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj"] Jul 24 00:41:48.631088 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:48.631066 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-eaba160e-gw-openshift-default-6dfcbfc68-w44rj"] Jul 24 00:41:50.001717 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:50.001682 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2154268-aa02-4a08-971e-62f73304f17f" path="/var/lib/kubelet/pods/a2154268-aa02-4a08-971e-62f73304f17f/volumes" Jul 24 00:41:52.583898 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:41:52.583860 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:42:00.579170 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:00.579132 2576 patch_prober.go:28] interesting pod/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 24 00:42:00.579636 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:00.579200 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 24 00:42:00.674400 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:00.674363 2576 generic.go:356] "Generic (PLEG): container finished" podID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerID="51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704" exitCode=0 Jul 24 00:42:00.674578 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:00.674432 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" event={"ID":"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6","Type":"ContainerDied","Data":"51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704"} Jul 24 00:42:00.674859 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:00.674846 2576 scope.go:117] "RemoveContainer" containerID="51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704" Jul 24 00:42:00.675889 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:00.675858 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:42:00.675959 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:00.675927 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:42:00.676005 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:00.675971 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:42:01.682101 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:01.682064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" event={"ID":"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6","Type":"ContainerStarted","Data":"b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0"} Jul 24 00:42:02.282841 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:02.282802 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:42:02.283104 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:02.283080 2576 patch_prober.go:28] interesting pod/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 24 00:42:02.283159 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:02.283128 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 24 00:42:03.283471 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:03.283432 2576 patch_prober.go:28] interesting pod/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 24 00:42:03.283983 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:03.283497 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 24 00:42:04.283586 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:04.283549 2576 patch_prober.go:28] interesting pod/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 24 00:42:04.284020 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:04.283608 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 24 00:42:05.283358 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:05.283316 2576 patch_prober.go:28] interesting pod/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 24 00:42:05.283570 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:05.283381 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 24 00:42:06.287536 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:06.287504 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:42:06.287972 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:06.287894 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:42:06.288676 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:06.288659 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:42:11.870406 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:11.870355 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv"] Jul 24 00:42:11.870805 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:11.870746 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2154268-aa02-4a08-971e-62f73304f17f" containerName="istio-proxy" Jul 24 00:42:11.870805 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:11.870763 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2154268-aa02-4a08-971e-62f73304f17f" containerName="istio-proxy" Jul 24 00:42:11.870805 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:11.870775 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2154268-aa02-4a08-971e-62f73304f17f" containerName="istio-proxy" Jul 24 00:42:11.870805 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:11.870780 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2154268-aa02-4a08-971e-62f73304f17f" containerName="istio-proxy" Jul 24 00:42:11.870942 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:11.870871 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2154268-aa02-4a08-971e-62f73304f17f" containerName="istio-proxy" Jul 24 00:42:11.874324 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:11.874303 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv" Jul 24 00:42:11.877035 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:11.877007 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 00:42:11.877976 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:11.877935 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\"" Jul 24 00:42:11.881471 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:11.881435 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv"] Jul 24 00:42:11.919929 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:11.919880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/964f250a-8d9b-4a8b-854a-e1e31a8e8527-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a227ad09-ccscv\" (UID: \"964f250a-8d9b-4a8b-854a-e1e31a8e8527\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv" Jul 24 00:42:12.021031 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:12.020990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/964f250a-8d9b-4a8b-854a-e1e31a8e8527-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a227ad09-ccscv\" (UID: \"964f250a-8d9b-4a8b-854a-e1e31a8e8527\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv" Jul 24 00:42:12.021644 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:12.021625 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/964f250a-8d9b-4a8b-854a-e1e31a8e8527-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a227ad09-ccscv\" (UID: \"964f250a-8d9b-4a8b-854a-e1e31a8e8527\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv" Jul 24 00:42:12.188321 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:12.188231 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv" Jul 24 00:42:12.330119 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:12.330088 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv"] Jul 24 00:42:12.331542 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:42:12.331510 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964f250a_8d9b_4a8b_854a_e1e31a8e8527.slice/crio-df6ed27d4b6bb8dc959a5ae65c368af0731af5279e6590f82a63e1bf3b9e73d6 WatchSource:0}: Error finding container df6ed27d4b6bb8dc959a5ae65c368af0731af5279e6590f82a63e1bf3b9e73d6: Status 404 returned error can't find the container with id df6ed27d4b6bb8dc959a5ae65c368af0731af5279e6590f82a63e1bf3b9e73d6 Jul 24 00:42:12.743574 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:12.743476 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv" event={"ID":"964f250a-8d9b-4a8b-854a-e1e31a8e8527","Type":"ContainerStarted","Data":"83b1a03cecff3623e9d97c310d9af2a99bedd3880b004b67b59c923447bd3b71"} Jul 24 00:42:12.743574 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:12.743558 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv" event={"ID":"964f250a-8d9b-4a8b-854a-e1e31a8e8527","Type":"ContainerStarted","Data":"df6ed27d4b6bb8dc959a5ae65c368af0731af5279e6590f82a63e1bf3b9e73d6"} Jul 24 00:42:12.759052 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:12.758997 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv" podStartSLOduration=1.758980547 podStartE2EDuration="1.758980547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:42:12.757692853 +0000 UTC m=+1709.330789656" watchObservedRunningTime="2026-07-24 00:42:12.758980547 +0000 UTC m=+1709.332077351" Jul 24 00:42:13.750341 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:13.750302 2576 generic.go:356] "Generic (PLEG): container finished" podID="964f250a-8d9b-4a8b-854a-e1e31a8e8527" containerID="83b1a03cecff3623e9d97c310d9af2a99bedd3880b004b67b59c923447bd3b71" exitCode=0 Jul 24 00:42:13.750727 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:13.750361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv" event={"ID":"964f250a-8d9b-4a8b-854a-e1e31a8e8527","Type":"ContainerDied","Data":"83b1a03cecff3623e9d97c310d9af2a99bedd3880b004b67b59c923447bd3b71"} Jul 24 00:42:14.911362 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:14.911333 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv" Jul 24 00:42:14.947229 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:14.947189 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/964f250a-8d9b-4a8b-854a-e1e31a8e8527-maas-api-service-ca\") pod \"964f250a-8d9b-4a8b-854a-e1e31a8e8527\" (UID: \"964f250a-8d9b-4a8b-854a-e1e31a8e8527\") " Jul 24 00:42:14.947662 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:14.947634 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964f250a-8d9b-4a8b-854a-e1e31a8e8527-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "964f250a-8d9b-4a8b-854a-e1e31a8e8527" (UID: "964f250a-8d9b-4a8b-854a-e1e31a8e8527"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:42:15.048082 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:15.048054 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/964f250a-8d9b-4a8b-854a-e1e31a8e8527-maas-api-service-ca\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:42:15.765374 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:15.765347 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv" Jul 24 00:42:15.765551 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:15.765386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv" event={"ID":"964f250a-8d9b-4a8b-854a-e1e31a8e8527","Type":"ContainerDied","Data":"df6ed27d4b6bb8dc959a5ae65c368af0731af5279e6590f82a63e1bf3b9e73d6"} Jul 24 00:42:15.765551 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:15.765430 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6ed27d4b6bb8dc959a5ae65c368af0731af5279e6590f82a63e1bf3b9e73d6" Jul 24 00:42:25.291591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:25.291547 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:42:26.945583 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:26.945541 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv"] Jul 24 00:42:26.947869 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:26.947844 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a227ad09-ccscv"] Jul 24 00:42:28.003220 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:28.003185 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="964f250a-8d9b-4a8b-854a-e1e31a8e8527" path="/var/lib/kubelet/pods/964f250a-8d9b-4a8b-854a-e1e31a8e8527/volumes" Jul 24 00:42:28.205197 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:28.205156 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn"] Jul 24 00:42:28.205416 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:28.205389 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerName="istio-proxy" containerID="cri-o://b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0" gracePeriod=30 Jul 24 00:42:29.043116 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.043071 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l"] Jul 24 00:42:29.043662 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.043627 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="964f250a-8d9b-4a8b-854a-e1e31a8e8527" containerName="revoke-keys" Jul 24 00:42:29.043662 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.043654 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="964f250a-8d9b-4a8b-854a-e1e31a8e8527" containerName="revoke-keys" Jul 24 00:42:29.043876 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.043775 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2154268-aa02-4a08-971e-62f73304f17f" containerName="istio-proxy" Jul 24 00:42:29.043876 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.043787 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="964f250a-8d9b-4a8b-854a-e1e31a8e8527" containerName="revoke-keys" Jul 24 00:42:29.047297 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.047275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.049745 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.049729 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-601239cb-openshift-default-dockercfg-d7hj2\"" Jul 24 00:42:29.057739 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.057708 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l"] Jul 24 00:42:29.175894 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.175855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-token\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.176109 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.175911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-workload-certs\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.176109 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.175943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-credential-socket\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.176109 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.175984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-envoy\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.176109 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.176019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4plx\" (UniqueName: \"kubernetes.io/projected/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-kube-api-access-n4plx\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.176109 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.176042 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-workload-socket\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.176109 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.176060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-data\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.176308 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.176104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-podinfo\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.176308 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.176133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istiod-ca-cert\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.277253 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-workload-socket\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.277253 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-data\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.277477 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-podinfo\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.277477 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istiod-ca-cert\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.277477 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-token\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.277620 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-workload-certs\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.277620 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-credential-socket\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.277620 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-envoy\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.277620 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4plx\" (UniqueName: \"kubernetes.io/projected/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-kube-api-access-n4plx\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.277861 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277696 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-workload-socket\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.277861 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277750 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-data\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.277861 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277824 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-credential-socket\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.278088 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.277879 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-workload-certs\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.278352 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.278335 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istiod-ca-cert\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.279727 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.279706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-envoy\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.279865 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.279848 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-podinfo\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.285858 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.285827 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-token\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.286014 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.285994 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4plx\" (UniqueName: \"kubernetes.io/projected/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-kube-api-access-n4plx\") pod \"e2e-derive-601239cb-openshift-default-645fb47648-wfr5l\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.359712 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.359617 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:29.510118 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.510085 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l"] Jul 24 00:42:29.510810 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:42:29.510782 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44a54d7_3c70_4ec8_a74f_10c063d5d5ef.slice/crio-f54159e5ae6ec433153427821af289a141c60d7ebcac77c24d464481ee071346 WatchSource:0}: Error finding container f54159e5ae6ec433153427821af289a141c60d7ebcac77c24d464481ee071346: Status 404 returned error can't find the container with id f54159e5ae6ec433153427821af289a141c60d7ebcac77c24d464481ee071346 Jul 24 00:42:29.513191 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.513160 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:42:29.513261 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.513221 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:42:29.513261 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.513250 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:42:29.846837 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.846803 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" event={"ID":"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef","Type":"ContainerStarted","Data":"68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf"} Jul 24 00:42:29.847073 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.846899 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" event={"ID":"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef","Type":"ContainerStarted","Data":"f54159e5ae6ec433153427821af289a141c60d7ebcac77c24d464481ee071346"} Jul 24 00:42:29.867021 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:29.866969 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" podStartSLOduration=0.866928004 podStartE2EDuration="866.928004ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:42:29.865635089 +0000 UTC m=+1726.438731913" watchObservedRunningTime="2026-07-24 00:42:29.866928004 +0000 UTC m=+1726.440024807" Jul 24 00:42:30.360826 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:30.360780 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:30.365989 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:30.365941 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:30.852217 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:30.852182 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:30.853441 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:30.853416 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:33.464556 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.464478 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:42:33.626229 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626174 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-podinfo\") pod \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " Jul 24 00:42:33.626448 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626256 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istiod-ca-cert\") pod \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " Jul 24 00:42:33.626448 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626278 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-credential-socket\") pod \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " Jul 24 00:42:33.626448 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626312 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-workload-certs\") pod \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " Jul 24 00:42:33.626448 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626329 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-data\") pod \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " Jul 24 00:42:33.626448 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626354 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-token\") pod \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " Jul 24 00:42:33.626448 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626376 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvpgf\" (UniqueName: \"kubernetes.io/projected/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-kube-api-access-cvpgf\") pod \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " Jul 24 00:42:33.626448 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626441 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-envoy\") pod \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " Jul 24 00:42:33.626830 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626483 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-workload-socket\") pod \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\" (UID: \"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6\") " Jul 24 00:42:33.626830 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626623 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" (UID: "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:42:33.626830 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626639 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" (UID: "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:42:33.626830 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626725 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" (UID: "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:42:33.627019 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626830 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" (UID: "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:42:33.627019 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626864 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:42:33.627019 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626884 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:42:33.627019 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.626898 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:42:33.627663 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.627634 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" (UID: "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:42:33.628571 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.628539 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" (UID: "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:42:33.628665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.628612 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" (UID: "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 00:42:33.628665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.628649 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-kube-api-access-cvpgf" (OuterVolumeSpecName: "kube-api-access-cvpgf") pod "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" (UID: "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6"). InnerVolumeSpecName "kube-api-access-cvpgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:42:33.628743 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.628688 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" (UID: "68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:42:33.727698 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.727605 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:42:33.727698 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.727639 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvpgf\" (UniqueName: \"kubernetes.io/projected/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-kube-api-access-cvpgf\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:42:33.727698 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.727649 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:42:33.727698 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.727660 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:42:33.727698 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.727668 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:42:33.727698 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.727676 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:42:33.870301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.870268 2576 generic.go:356] "Generic (PLEG): container finished" podID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerID="b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0" exitCode=0 Jul 24 00:42:33.870482 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.870317 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" Jul 24 00:42:33.870482 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.870350 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" event={"ID":"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6","Type":"ContainerDied","Data":"b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0"} Jul 24 00:42:33.870482 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.870404 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn" event={"ID":"68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6","Type":"ContainerDied","Data":"24f9e6a815012b0cfad088ee32ff86200ba87855ecc03db09e42f3ad06223868"} Jul 24 00:42:33.870482 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.870427 2576 scope.go:117] "RemoveContainer" containerID="b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0" Jul 24 00:42:33.881130 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.881107 2576 scope.go:117] "RemoveContainer" containerID="51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704" Jul 24 00:42:33.890457 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.890427 2576 scope.go:117] "RemoveContainer" containerID="b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0" Jul 24 00:42:33.890891 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:42:33.890859 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0\": container with ID starting with b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0 not found: ID does not exist" containerID="b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0" Jul 24 00:42:33.891035 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.890902 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0"} err="failed to get container status \"b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0\": rpc error: code = NotFound desc = could not find container \"b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0\": container with ID starting with b75f58e23584ba6bda81b65bc09afffcc0cefa95688d727d7e8e47b7fccdddd0 not found: ID does not exist" Jul 24 00:42:33.891035 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.890928 2576 scope.go:117] "RemoveContainer" containerID="51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704" Jul 24 00:42:33.891334 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:42:33.891308 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704\": container with ID starting with 51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704 not found: ID does not exist" containerID="51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704" Jul 24 00:42:33.891411 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.891345 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704"} err="failed to get container status \"51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704\": rpc error: code = NotFound desc = could not find container \"51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704\": container with ID starting with 51bd48eab4a318bea728e50f8e97405abae160a407b9fa889d7f1a638ba73704 not found: ID does not exist" Jul 24 00:42:33.893699 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.893675 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn"] Jul 24 00:42:33.897403 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:33.897370 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a227ad09-openshift-default-699b7f998d-xctrn"] Jul 24 00:42:34.002229 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:34.002148 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" path="/var/lib/kubelet/pods/68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6/volumes" Jul 24 00:42:36.886646 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:36.886607 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:42:45.853683 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:45.853639 2576 patch_prober.go:28] interesting pod/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 24 00:42:45.854152 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:45.853737 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 24 00:42:45.942889 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:45.942852 2576 generic.go:356] "Generic (PLEG): container finished" podID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerID="68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf" exitCode=0 Jul 24 00:42:45.943106 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:45.942927 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" event={"ID":"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef","Type":"ContainerDied","Data":"68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf"} Jul 24 00:42:45.943427 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:45.943407 2576 scope.go:117] "RemoveContainer" containerID="68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf" Jul 24 00:42:45.944526 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:45.944497 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:42:45.944598 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:45.944581 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:42:45.944650 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:45.944633 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:42:46.956649 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:46.956596 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" event={"ID":"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef","Type":"ContainerStarted","Data":"37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c"} Jul 24 00:42:47.360682 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:47.360638 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:47.361017 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:47.360986 2576 patch_prober.go:28] interesting pod/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 24 00:42:47.361076 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:47.361053 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 24 00:42:48.360403 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:48.360357 2576 patch_prober.go:28] interesting pod/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 24 00:42:48.360809 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:48.360435 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 24 00:42:49.360282 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:49.360246 2576 patch_prober.go:28] interesting pod/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 24 00:42:49.360467 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:49.360303 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 24 00:42:50.360699 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:50.360646 2576 patch_prober.go:28] interesting pod/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 24 00:42:50.361140 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:50.360737 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 24 00:42:51.364194 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:51.364158 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:51.364595 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:51.364515 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:51.365206 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:51.365185 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:42:56.154332 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.154298 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75"] Jul 24 00:42:56.154755 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.154642 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerName="istio-proxy" Jul 24 00:42:56.154755 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.154652 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerName="istio-proxy" Jul 24 00:42:56.154755 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.154726 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerName="istio-proxy" Jul 24 00:42:56.158038 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.158022 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75" Jul 24 00:42:56.160301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.160284 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 00:42:56.160365 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.160289 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\"" Jul 24 00:42:56.164809 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.164778 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75"] Jul 24 00:42:56.220665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.220630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc405f-90aa-49b9-b283-ee077320c97c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-601239cb-rqp75\" (UID: \"8ecc405f-90aa-49b9-b283-ee077320c97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75" Jul 24 00:42:56.321551 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.321511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc405f-90aa-49b9-b283-ee077320c97c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-601239cb-rqp75\" (UID: \"8ecc405f-90aa-49b9-b283-ee077320c97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75" Jul 24 00:42:56.322167 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.322146 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc405f-90aa-49b9-b283-ee077320c97c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-601239cb-rqp75\" (UID: \"8ecc405f-90aa-49b9-b283-ee077320c97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75" Jul 24 00:42:56.469990 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.469869 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75" Jul 24 00:42:56.604503 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:56.604449 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75"] Jul 24 00:42:56.605699 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:42:56.605655 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecc405f_90aa_49b9_b283_ee077320c97c.slice/crio-65b8313a04966c9e7cdf92b4653f5a2c38e4bfe83480fbe66765f44e7a4dfbdc WatchSource:0}: Error finding container 65b8313a04966c9e7cdf92b4653f5a2c38e4bfe83480fbe66765f44e7a4dfbdc: Status 404 returned error can't find the container with id 65b8313a04966c9e7cdf92b4653f5a2c38e4bfe83480fbe66765f44e7a4dfbdc Jul 24 00:42:57.013323 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:57.013292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75" event={"ID":"8ecc405f-90aa-49b9-b283-ee077320c97c","Type":"ContainerStarted","Data":"8f7d310875881ff2e55b80d7d22d9447afa1db453a9b9c725db7916b96d46475"} Jul 24 00:42:57.013493 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:57.013352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75" event={"ID":"8ecc405f-90aa-49b9-b283-ee077320c97c","Type":"ContainerStarted","Data":"65b8313a04966c9e7cdf92b4653f5a2c38e4bfe83480fbe66765f44e7a4dfbdc"} Jul 24 00:42:57.029923 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:57.029873 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75" podStartSLOduration=1.029857926 podStartE2EDuration="1.029857926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:42:57.028626808 +0000 UTC m=+1753.601723623" watchObservedRunningTime="2026-07-24 00:42:57.029857926 +0000 UTC m=+1753.602954764" Jul 24 00:42:58.019911 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:58.019873 2576 generic.go:356] "Generic (PLEG): container finished" podID="8ecc405f-90aa-49b9-b283-ee077320c97c" containerID="8f7d310875881ff2e55b80d7d22d9447afa1db453a9b9c725db7916b96d46475" exitCode=0 Jul 24 00:42:58.020352 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:58.019969 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75" event={"ID":"8ecc405f-90aa-49b9-b283-ee077320c97c","Type":"ContainerDied","Data":"8f7d310875881ff2e55b80d7d22d9447afa1db453a9b9c725db7916b96d46475"} Jul 24 00:42:59.163319 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:59.163289 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75" Jul 24 00:42:59.252038 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:59.251995 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc405f-90aa-49b9-b283-ee077320c97c-maas-api-service-ca\") pod \"8ecc405f-90aa-49b9-b283-ee077320c97c\" (UID: \"8ecc405f-90aa-49b9-b283-ee077320c97c\") " Jul 24 00:42:59.252270 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:59.252250 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecc405f-90aa-49b9-b283-ee077320c97c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ecc405f-90aa-49b9-b283-ee077320c97c" (UID: "8ecc405f-90aa-49b9-b283-ee077320c97c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:42:59.353208 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:42:59.353172 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc405f-90aa-49b9-b283-ee077320c97c-maas-api-service-ca\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:43:00.032163 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:00.032135 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75" Jul 24 00:43:00.032378 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:00.032131 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75" event={"ID":"8ecc405f-90aa-49b9-b283-ee077320c97c","Type":"ContainerDied","Data":"65b8313a04966c9e7cdf92b4653f5a2c38e4bfe83480fbe66765f44e7a4dfbdc"} Jul 24 00:43:00.032378 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:00.032258 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65b8313a04966c9e7cdf92b4653f5a2c38e4bfe83480fbe66765f44e7a4dfbdc" Jul 24 00:43:09.783902 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:09.783862 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:43:11.235114 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:11.235076 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75"] Jul 24 00:43:11.238205 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:11.238176 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-601239cb-rqp75"] Jul 24 00:43:11.367472 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:11.367399 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l"] Jul 24 00:43:11.367767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:11.367710 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerName="istio-proxy" containerID="cri-o://37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c" gracePeriod=30 Jul 24 00:43:12.004003 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:12.003967 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ecc405f-90aa-49b9-b283-ee077320c97c" path="/var/lib/kubelet/pods/8ecc405f-90aa-49b9-b283-ee077320c97c/volumes" Jul 24 00:43:16.622009 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.621982 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:43:16.719099 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719009 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-workload-certs\") pod \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " Jul 24 00:43:16.719099 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719041 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-podinfo\") pod \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " Jul 24 00:43:16.719099 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719090 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4plx\" (UniqueName: \"kubernetes.io/projected/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-kube-api-access-n4plx\") pod \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " Jul 24 00:43:16.719353 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719141 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-credential-socket\") pod \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " Jul 24 00:43:16.719353 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719175 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istiod-ca-cert\") pod \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " Jul 24 00:43:16.719353 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719220 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-token\") pod \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " Jul 24 00:43:16.719353 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719241 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-envoy\") pod \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " Jul 24 00:43:16.719353 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719270 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-workload-socket\") pod \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " Jul 24 00:43:16.719353 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719321 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" (UID: "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:43:16.719671 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719340 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-data\") pod \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\" (UID: \"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef\") " Jul 24 00:43:16.719671 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719548 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" (UID: "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:43:16.719778 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719686 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" (UID: "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:43:16.719778 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719729 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" (UID: "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:43:16.719778 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719747 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:43:16.719778 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.719766 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:43:16.720526 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.720496 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" (UID: "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:43:16.721411 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.721377 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" (UID: "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:43:16.721411 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.721403 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" (UID: "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 00:43:16.721562 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.721512 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" (UID: "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:43:16.721603 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.721555 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-kube-api-access-n4plx" (OuterVolumeSpecName: "kube-api-access-n4plx") pod "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" (UID: "c44a54d7-3c70-4ec8-a74f-10c063d5d5ef"). InnerVolumeSpecName "kube-api-access-n4plx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:43:16.820938 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.820896 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:43:16.820938 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.820932 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:43:16.820938 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.820977 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:43:16.821214 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.820990 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:43:16.821214 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.821001 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:43:16.821214 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.821011 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:43:16.821214 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:16.821024 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4plx\" (UniqueName: \"kubernetes.io/projected/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef-kube-api-access-n4plx\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:43:17.135181 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:17.135148 2576 generic.go:356] "Generic (PLEG): container finished" podID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerID="37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c" exitCode=0 Jul 24 00:43:17.135361 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:17.135192 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" Jul 24 00:43:17.135361 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:17.135239 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" event={"ID":"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef","Type":"ContainerDied","Data":"37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c"} Jul 24 00:43:17.135361 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:17.135298 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l" event={"ID":"c44a54d7-3c70-4ec8-a74f-10c063d5d5ef","Type":"ContainerDied","Data":"f54159e5ae6ec433153427821af289a141c60d7ebcac77c24d464481ee071346"} Jul 24 00:43:17.135361 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:17.135323 2576 scope.go:117] "RemoveContainer" containerID="37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c" Jul 24 00:43:17.145186 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:17.145165 2576 scope.go:117] "RemoveContainer" containerID="68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf" Jul 24 00:43:17.153648 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:17.153631 2576 scope.go:117] "RemoveContainer" containerID="37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c" Jul 24 00:43:17.153921 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:43:17.153902 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c\": container with ID starting with 37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c not found: ID does not exist" containerID="37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c" Jul 24 00:43:17.154026 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:17.153928 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c"} err="failed to get container status \"37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c\": rpc error: code = NotFound desc = could not find container \"37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c\": container with ID starting with 37061c77414aada68dab7b8308df375398f7631022a4ec1b2cff7a510f76971c not found: ID does not exist" Jul 24 00:43:17.154448 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:17.154436 2576 scope.go:117] "RemoveContainer" containerID="68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf" Jul 24 00:43:17.154675 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:43:17.154658 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf\": container with ID starting with 68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf not found: ID does not exist" containerID="68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf" Jul 24 00:43:17.154738 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:17.154679 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf"} err="failed to get container status \"68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf\": rpc error: code = NotFound desc = could not find container \"68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf\": container with ID starting with 68cccc1783a2bfa6068dbd3b239ac013fdb8aab73433b28ef3b20c2b19aafaaf not found: ID does not exist" Jul 24 00:43:17.157875 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:17.157849 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l"] Jul 24 00:43:17.161902 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:17.161883 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-601239cb-openshift-default-645fb47648-wfr5l"] Jul 24 00:43:18.002326 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:18.002291 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" path="/var/lib/kubelet/pods/c44a54d7-3c70-4ec8-a74f-10c063d5d5ef/volumes" Jul 24 00:43:44.051511 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:44.051479 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:43:44.057136 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:43:44.057106 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:44:35.290255 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:35.290212 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:44:46.584425 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:46.584379 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:44:48.431739 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.431701 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x"] Jul 24 00:44:48.432257 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.432237 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerName="istio-proxy" Jul 24 00:44:48.432349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.432260 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerName="istio-proxy" Jul 24 00:44:48.432349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.432284 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerName="istio-proxy" Jul 24 00:44:48.432349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.432293 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerName="istio-proxy" Jul 24 00:44:48.432349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.432303 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerName="istio-proxy" Jul 24 00:44:48.432349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.432311 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerName="istio-proxy" Jul 24 00:44:48.432349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.432324 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecc405f-90aa-49b9-b283-ee077320c97c" containerName="revoke-keys" Jul 24 00:44:48.432349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.432333 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecc405f-90aa-49b9-b283-ee077320c97c" containerName="revoke-keys" Jul 24 00:44:48.432683 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.432435 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerName="istio-proxy" Jul 24 00:44:48.432683 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.432450 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecc405f-90aa-49b9-b283-ee077320c97c" containerName="revoke-keys" Jul 24 00:44:48.432683 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.432462 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f3d507-4a11-4e04-bf9c-ecf4b1d4e8c6" containerName="istio-proxy" Jul 24 00:44:48.435723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.435701 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.438046 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.438022 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-587b8d23-openshift-default-dockercfg-8fz5r\"" Jul 24 00:44:48.456030 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.455993 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x"] Jul 24 00:44:48.530984 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.530902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-credential-socket\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.531195 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.531083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-data\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.531195 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.531121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwxms\" (UniqueName: \"kubernetes.io/projected/ec3d947b-f999-40ef-b087-c53c673e5ce0-kube-api-access-zwxms\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.531195 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.531166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-envoy\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.531380 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.531219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-podinfo\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.531380 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.531256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3d947b-f999-40ef-b087-c53c673e5ce0-istiod-ca-cert\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.531380 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.531298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-workload-socket\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.531380 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.531323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-workload-certs\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.531380 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.531359 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-token\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.632605 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.632568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-data\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.632605 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.632606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwxms\" (UniqueName: \"kubernetes.io/projected/ec3d947b-f999-40ef-b087-c53c673e5ce0-kube-api-access-zwxms\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.632858 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.632637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-envoy\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.632858 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.632667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-podinfo\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.632858 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.632701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3d947b-f999-40ef-b087-c53c673e5ce0-istiod-ca-cert\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.632858 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.632835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-workload-socket\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.633103 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.632878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-workload-certs\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.633103 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.632919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-token\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.633103 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.633008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-credential-socket\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.633103 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.633089 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-data\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.633304 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.633209 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-workload-socket\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.633304 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.633256 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-workload-certs\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.633507 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.633477 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-credential-socket\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.633507 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.633491 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3d947b-f999-40ef-b087-c53c673e5ce0-istiod-ca-cert\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.634991 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.634967 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-envoy\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.635351 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.635332 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-podinfo\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.651418 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.651384 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-token\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.651565 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.651518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwxms\" (UniqueName: \"kubernetes.io/projected/ec3d947b-f999-40ef-b087-c53c673e5ce0-kube-api-access-zwxms\") pod \"e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.746727 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.746621 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:48.889976 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.889931 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x"] Jul 24 00:44:48.890844 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:44:48.890822 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3d947b_f999_40ef_b087_c53c673e5ce0.slice/crio-900a0ae663a696b82d8c3ec36237e37d3fe88c489e804a9623fd0919fe5c5d6b WatchSource:0}: Error finding container 900a0ae663a696b82d8c3ec36237e37d3fe88c489e804a9623fd0919fe5c5d6b: Status 404 returned error can't find the container with id 900a0ae663a696b82d8c3ec36237e37d3fe88c489e804a9623fd0919fe5c5d6b Jul 24 00:44:48.892976 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.892928 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:44:48.893082 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.893010 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:44:48.893082 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:48.893041 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:44:49.674893 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:49.674855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" event={"ID":"ec3d947b-f999-40ef-b087-c53c673e5ce0","Type":"ContainerStarted","Data":"3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4"} Jul 24 00:44:49.675329 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:49.674911 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" event={"ID":"ec3d947b-f999-40ef-b087-c53c673e5ce0","Type":"ContainerStarted","Data":"900a0ae663a696b82d8c3ec36237e37d3fe88c489e804a9623fd0919fe5c5d6b"} Jul 24 00:44:49.697278 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:49.697222 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" podStartSLOduration=1.697205196 podStartE2EDuration="1.697205196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:44:49.6952488 +0000 UTC m=+1866.268345643" watchObservedRunningTime="2026-07-24 00:44:49.697205196 +0000 UTC m=+1866.270301999" Jul 24 00:44:49.747563 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:49.747532 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:49.752603 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:49.752575 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:50.681060 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:50.681033 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:50.682155 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:50.682125 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:44:58.588867 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:44:58.588829 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:45:00.131018 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.130973 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv"] Jul 24 00:45:00.131644 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.131624 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c44a54d7-3c70-4ec8-a74f-10c063d5d5ef" containerName="istio-proxy" Jul 24 00:45:00.136207 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.136175 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747565-4pp8v"] Jul 24 00:45:00.138045 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.138017 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv" Jul 24 00:45:00.139999 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.139974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747565-4pp8v" Jul 24 00:45:00.142332 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.142300 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\"" Jul 24 00:45:00.144229 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.144201 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv"] Jul 24 00:45:00.148583 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.148555 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747565-4pp8v"] Jul 24 00:45:00.164338 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.164313 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv" Jul 24 00:45:00.175980 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.175925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747565-4pp8v" Jul 24 00:45:00.366100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.362714 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv"] Jul 24 00:45:00.401443 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.401412 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747565-4pp8v"] Jul 24 00:45:00.403382 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:45:00.403354 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd13eda_c959_4950_b88d_c70b141fb37f.slice/crio-640bb9ff55642d0c609e684dc56d68ecb3c40bec886e9d34f091c9160b3b7623 WatchSource:0}: Error finding container 640bb9ff55642d0c609e684dc56d68ecb3c40bec886e9d34f091c9160b3b7623: Status 404 returned error can't find the container with id 640bb9ff55642d0c609e684dc56d68ecb3c40bec886e9d34f091c9160b3b7623 Jul 24 00:45:00.747294 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.747243 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv" event={"ID":"3176b495-c5ee-4a88-b57a-db7206c50c27","Type":"ContainerStarted","Data":"8b8417dfa90f0923a6ec70ccdd4e079900caed08469632aeda673fcd51c73213"} Jul 24 00:45:00.747503 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.747326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv" event={"ID":"3176b495-c5ee-4a88-b57a-db7206c50c27","Type":"ContainerStarted","Data":"c6999d72c6c0b859d154002a8d5ad27e69d7afcc2e947fb0840a1cd7eac9a626"} Jul 24 00:45:00.750075 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.750039 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747565-4pp8v" event={"ID":"ffd13eda-c959-4950-b88d-c70b141fb37f","Type":"ContainerStarted","Data":"52509dd51d603ed4dacdef08ff7fd0eedb2741e95b8f23c7349c0193480a55d2"} Jul 24 00:45:00.750251 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.750100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747565-4pp8v" event={"ID":"ffd13eda-c959-4950-b88d-c70b141fb37f","Type":"ContainerStarted","Data":"640bb9ff55642d0c609e684dc56d68ecb3c40bec886e9d34f091c9160b3b7623"} Jul 24 00:45:00.780492 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.780428 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv" podStartSLOduration=0.780407261 podStartE2EDuration="780.407261ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:45:00.778999239 +0000 UTC m=+1877.352096045" watchObservedRunningTime="2026-07-24 00:45:00.780407261 +0000 UTC m=+1877.353504063" Jul 24 00:45:00.812162 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:00.812106 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747565-4pp8v" podStartSLOduration=0.812091176 podStartE2EDuration="812.091176ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:45:00.81000453 +0000 UTC m=+1877.383101333" watchObservedRunningTime="2026-07-24 00:45:00.812091176 +0000 UTC m=+1877.385188036" Jul 24 00:45:01.764769 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:01.764723 2576 generic.go:356] "Generic (PLEG): container finished" podID="3176b495-c5ee-4a88-b57a-db7206c50c27" containerID="8b8417dfa90f0923a6ec70ccdd4e079900caed08469632aeda673fcd51c73213" exitCode=7 Jul 24 00:45:01.765381 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:01.764906 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv" event={"ID":"3176b495-c5ee-4a88-b57a-db7206c50c27","Type":"ContainerDied","Data":"8b8417dfa90f0923a6ec70ccdd4e079900caed08469632aeda673fcd51c73213"} Jul 24 00:45:01.765523 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:01.765012 2576 scope.go:117] "RemoveContainer" containerID="8b8417dfa90f0923a6ec70ccdd4e079900caed08469632aeda673fcd51c73213" Jul 24 00:45:02.772591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:02.772556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv" event={"ID":"3176b495-c5ee-4a88-b57a-db7206c50c27","Type":"ContainerStarted","Data":"ef3268de8b12bd630c1afa7e8fd0c5cfc676b7aaf73ae16b6398dee9589fe74c"} Jul 24 00:45:02.774184 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:02.774155 2576 generic.go:356] "Generic (PLEG): container finished" podID="ffd13eda-c959-4950-b88d-c70b141fb37f" containerID="52509dd51d603ed4dacdef08ff7fd0eedb2741e95b8f23c7349c0193480a55d2" exitCode=0 Jul 24 00:45:02.774320 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:02.774212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747565-4pp8v" event={"ID":"ffd13eda-c959-4950-b88d-c70b141fb37f","Type":"ContainerDied","Data":"52509dd51d603ed4dacdef08ff7fd0eedb2741e95b8f23c7349c0193480a55d2"} Jul 24 00:45:03.443258 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.443216 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp"] Jul 24 00:45:03.446679 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.446652 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp" Jul 24 00:45:03.449178 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.449155 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 00:45:03.454576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.454460 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp"] Jul 24 00:45:03.591641 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.591594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4475610-1280-485a-8c3f-ea7324fb9d9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp\" (UID: \"d4475610-1280-485a-8c3f-ea7324fb9d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp" Jul 24 00:45:03.692874 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.692834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4475610-1280-485a-8c3f-ea7324fb9d9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp\" (UID: \"d4475610-1280-485a-8c3f-ea7324fb9d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp" Jul 24 00:45:03.693529 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.693508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4475610-1280-485a-8c3f-ea7324fb9d9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp\" (UID: \"d4475610-1280-485a-8c3f-ea7324fb9d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp" Jul 24 00:45:03.759496 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.759459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp" Jul 24 00:45:03.787808 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.786544 2576 generic.go:356] "Generic (PLEG): container finished" podID="3176b495-c5ee-4a88-b57a-db7206c50c27" containerID="ef3268de8b12bd630c1afa7e8fd0c5cfc676b7aaf73ae16b6398dee9589fe74c" exitCode=0 Jul 24 00:45:03.787808 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.787006 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv" event={"ID":"3176b495-c5ee-4a88-b57a-db7206c50c27","Type":"ContainerDied","Data":"ef3268de8b12bd630c1afa7e8fd0c5cfc676b7aaf73ae16b6398dee9589fe74c"} Jul 24 00:45:03.787808 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.787121 2576 scope.go:117] "RemoveContainer" containerID="8b8417dfa90f0923a6ec70ccdd4e079900caed08469632aeda673fcd51c73213" Jul 24 00:45:03.933929 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.933902 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp"] Jul 24 00:45:03.936436 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:45:03.936404 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4475610_1280_485a_8c3f_ea7324fb9d9a.slice/crio-e5518f1ce42af9e4df822ffbf8ec5a476b38db265f96f1326416e3a17bd577ff WatchSource:0}: Error finding container e5518f1ce42af9e4df822ffbf8ec5a476b38db265f96f1326416e3a17bd577ff: Status 404 returned error can't find the container with id e5518f1ce42af9e4df822ffbf8ec5a476b38db265f96f1326416e3a17bd577ff Jul 24 00:45:03.944845 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:03.944822 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747565-4pp8v" Jul 24 00:45:04.796661 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:04.796622 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4475610-1280-485a-8c3f-ea7324fb9d9a" containerID="284b64e21e71cedd0b7ad9f98d713f2bbbef96a5829960b5143e171a9966179c" exitCode=0 Jul 24 00:45:04.797061 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:04.796692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp" event={"ID":"d4475610-1280-485a-8c3f-ea7324fb9d9a","Type":"ContainerDied","Data":"284b64e21e71cedd0b7ad9f98d713f2bbbef96a5829960b5143e171a9966179c"} Jul 24 00:45:04.797061 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:04.796757 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp" event={"ID":"d4475610-1280-485a-8c3f-ea7324fb9d9a","Type":"ContainerStarted","Data":"e5518f1ce42af9e4df822ffbf8ec5a476b38db265f96f1326416e3a17bd577ff"} Jul 24 00:45:04.798301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:04.798281 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747565-4pp8v" Jul 24 00:45:04.798459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:04.798309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747565-4pp8v" event={"ID":"ffd13eda-c959-4950-b88d-c70b141fb37f","Type":"ContainerDied","Data":"640bb9ff55642d0c609e684dc56d68ecb3c40bec886e9d34f091c9160b3b7623"} Jul 24 00:45:04.798459 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:04.798361 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="640bb9ff55642d0c609e684dc56d68ecb3c40bec886e9d34f091c9160b3b7623" Jul 24 00:45:04.942679 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:04.942654 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv" Jul 24 00:45:05.807272 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:05.807239 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv" Jul 24 00:45:05.807272 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:05.807253 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv" event={"ID":"3176b495-c5ee-4a88-b57a-db7206c50c27","Type":"ContainerDied","Data":"c6999d72c6c0b859d154002a8d5ad27e69d7afcc2e947fb0840a1cd7eac9a626"} Jul 24 00:45:05.807744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:05.807353 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6999d72c6c0b859d154002a8d5ad27e69d7afcc2e947fb0840a1cd7eac9a626" Jul 24 00:45:05.951467 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:05.951441 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp" Jul 24 00:45:06.114477 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.114385 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4475610-1280-485a-8c3f-ea7324fb9d9a-maas-api-service-ca\") pod \"d4475610-1280-485a-8c3f-ea7324fb9d9a\" (UID: \"d4475610-1280-485a-8c3f-ea7324fb9d9a\") " Jul 24 00:45:06.114821 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.114796 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4475610-1280-485a-8c3f-ea7324fb9d9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4475610-1280-485a-8c3f-ea7324fb9d9a" (UID: "d4475610-1280-485a-8c3f-ea7324fb9d9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:45:06.216029 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.215980 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4475610-1280-485a-8c3f-ea7324fb9d9a-maas-api-service-ca\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:45:06.427880 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.427373 2576 patch_prober.go:28] interesting pod/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:55006->10.133.0.54:15021: read: connection reset by peer" start-of-body= Jul 24 00:45:06.427880 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.427476 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:55006->10.133.0.54:15021: read: connection reset by peer" Jul 24 00:45:06.815005 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.814969 2576 generic.go:356] "Generic (PLEG): container finished" podID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerID="3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4" exitCode=0 Jul 24 00:45:06.815005 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.814987 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" event={"ID":"ec3d947b-f999-40ef-b087-c53c673e5ce0","Type":"ContainerDied","Data":"3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4"} Jul 24 00:45:06.815506 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.815487 2576 scope.go:117] "RemoveContainer" containerID="3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4" Jul 24 00:45:06.816645 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.816618 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp" event={"ID":"d4475610-1280-485a-8c3f-ea7324fb9d9a","Type":"ContainerDied","Data":"e5518f1ce42af9e4df822ffbf8ec5a476b38db265f96f1326416e3a17bd577ff"} Jul 24 00:45:06.816734 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.816634 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:45:06.816734 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.816665 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5518f1ce42af9e4df822ffbf8ec5a476b38db265f96f1326416e3a17bd577ff" Jul 24 00:45:06.816734 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.816633 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp" Jul 24 00:45:06.816734 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.816702 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:45:06.816870 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:06.816740 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:45:07.824840 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:07.824804 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" event={"ID":"ec3d947b-f999-40ef-b087-c53c673e5ce0","Type":"ContainerStarted","Data":"277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4"} Jul 24 00:45:08.748032 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:08.747989 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:45:08.748257 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:08.748151 2576 patch_prober.go:28] interesting pod/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 24 00:45:08.748257 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:08.748199 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 24 00:45:09.748175 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:09.748133 2576 patch_prober.go:28] interesting pod/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 24 00:45:09.748799 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:09.748763 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 24 00:45:10.747461 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:10.747409 2576 patch_prober.go:28] interesting pod/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 24 00:45:10.747683 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:10.747483 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 24 00:45:11.748597 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:11.748112 2576 patch_prober.go:28] interesting pod/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 24 00:45:11.748597 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:11.748180 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 24 00:45:12.751404 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:12.751372 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:45:12.751879 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:12.751777 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:45:12.752576 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:12.752555 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:45:13.910537 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:13.910494 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv"] Jul 24 00:45:13.913062 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:13.913038 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-587b8d23-29747565-2s6dv"] Jul 24 00:45:14.003023 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:14.002988 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3176b495-c5ee-4a88-b57a-db7206c50c27" path="/var/lib/kubelet/pods/3176b495-c5ee-4a88-b57a-db7206c50c27/volumes" Jul 24 00:45:16.686538 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:16.686503 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:45:18.571716 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:18.571678 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp"] Jul 24 00:45:18.573910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:18.573888 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-587b8d23-5r4pp"] Jul 24 00:45:20.001686 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:20.001653 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4475610-1280-485a-8c3f-ea7324fb9d9a" path="/var/lib/kubelet/pods/d4475610-1280-485a-8c3f-ea7324fb9d9a/volumes" Jul 24 00:45:24.878064 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:24.878026 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x"] Jul 24 00:45:24.878506 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:24.878359 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" containerID="cri-o://277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4" gracePeriod=30 Jul 24 00:45:27.754339 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:27.754294 2576 patch_prober.go:28] interesting pod/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 00:45:27.754735 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:27.754369 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 00:45:30.148811 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.148750 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:45:30.247058 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247025 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-token\") pod \"ec3d947b-f999-40ef-b087-c53c673e5ce0\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " Jul 24 00:45:30.247058 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247059 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwxms\" (UniqueName: \"kubernetes.io/projected/ec3d947b-f999-40ef-b087-c53c673e5ce0-kube-api-access-zwxms\") pod \"ec3d947b-f999-40ef-b087-c53c673e5ce0\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " Jul 24 00:45:30.247315 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247083 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-workload-socket\") pod \"ec3d947b-f999-40ef-b087-c53c673e5ce0\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " Jul 24 00:45:30.247315 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247114 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-credential-socket\") pod \"ec3d947b-f999-40ef-b087-c53c673e5ce0\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " Jul 24 00:45:30.247315 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247222 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-workload-certs\") pod \"ec3d947b-f999-40ef-b087-c53c673e5ce0\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " Jul 24 00:45:30.247315 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247296 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-data\") pod \"ec3d947b-f999-40ef-b087-c53c673e5ce0\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " Jul 24 00:45:30.247532 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247343 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec3d947b-f999-40ef-b087-c53c673e5ce0" (UID: "ec3d947b-f999-40ef-b087-c53c673e5ce0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:45:30.247532 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247359 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3d947b-f999-40ef-b087-c53c673e5ce0-istiod-ca-cert\") pod \"ec3d947b-f999-40ef-b087-c53c673e5ce0\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " Jul 24 00:45:30.247532 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247439 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec3d947b-f999-40ef-b087-c53c673e5ce0" (UID: "ec3d947b-f999-40ef-b087-c53c673e5ce0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:45:30.247532 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247467 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-podinfo\") pod \"ec3d947b-f999-40ef-b087-c53c673e5ce0\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " Jul 24 00:45:30.247532 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247517 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-envoy\") pod \"ec3d947b-f999-40ef-b087-c53c673e5ce0\" (UID: \"ec3d947b-f999-40ef-b087-c53c673e5ce0\") " Jul 24 00:45:30.247799 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247595 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec3d947b-f999-40ef-b087-c53c673e5ce0" (UID: "ec3d947b-f999-40ef-b087-c53c673e5ce0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:45:30.247799 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247704 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3d947b-f999-40ef-b087-c53c673e5ce0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec3d947b-f999-40ef-b087-c53c673e5ce0" (UID: "ec3d947b-f999-40ef-b087-c53c673e5ce0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:45:30.247908 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247847 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:45:30.247908 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247865 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:45:30.247908 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247879 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:45:30.247908 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.247894 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3d947b-f999-40ef-b087-c53c673e5ce0-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:45:30.248364 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.248337 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec3d947b-f999-40ef-b087-c53c673e5ce0" (UID: "ec3d947b-f999-40ef-b087-c53c673e5ce0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:45:30.249497 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.249471 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec3d947b-f999-40ef-b087-c53c673e5ce0" (UID: "ec3d947b-f999-40ef-b087-c53c673e5ce0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:45:30.249497 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.249490 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec3d947b-f999-40ef-b087-c53c673e5ce0" (UID: "ec3d947b-f999-40ef-b087-c53c673e5ce0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 00:45:30.249719 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.249701 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3d947b-f999-40ef-b087-c53c673e5ce0-kube-api-access-zwxms" (OuterVolumeSpecName: "kube-api-access-zwxms") pod "ec3d947b-f999-40ef-b087-c53c673e5ce0" (UID: "ec3d947b-f999-40ef-b087-c53c673e5ce0"). InnerVolumeSpecName "kube-api-access-zwxms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:45:30.249719 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.249709 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec3d947b-f999-40ef-b087-c53c673e5ce0" (UID: "ec3d947b-f999-40ef-b087-c53c673e5ce0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:45:30.348982 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.348911 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:45:30.348982 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.348974 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:45:30.348982 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.348983 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:45:30.348982 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.348992 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3d947b-f999-40ef-b087-c53c673e5ce0-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:45:30.349264 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.349001 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwxms\" (UniqueName: \"kubernetes.io/projected/ec3d947b-f999-40ef-b087-c53c673e5ce0-kube-api-access-zwxms\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:45:30.964239 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.964204 2576 generic.go:356] "Generic (PLEG): container finished" podID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerID="277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4" exitCode=0 Jul 24 00:45:30.964394 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.964271 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" Jul 24 00:45:30.964394 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.964290 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" event={"ID":"ec3d947b-f999-40ef-b087-c53c673e5ce0","Type":"ContainerDied","Data":"277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4"} Jul 24 00:45:30.964394 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.964374 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x" event={"ID":"ec3d947b-f999-40ef-b087-c53c673e5ce0","Type":"ContainerDied","Data":"900a0ae663a696b82d8c3ec36237e37d3fe88c489e804a9623fd0919fe5c5d6b"} Jul 24 00:45:30.964529 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.964397 2576 scope.go:117] "RemoveContainer" containerID="277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4" Jul 24 00:45:30.974529 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.974511 2576 scope.go:117] "RemoveContainer" containerID="3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4" Jul 24 00:45:30.987177 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.987148 2576 scope.go:117] "RemoveContainer" containerID="277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4" Jul 24 00:45:30.987472 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:45:30.987448 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4\": container with ID starting with 277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4 not found: ID does not exist" containerID="277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4" Jul 24 00:45:30.987577 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.987482 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4"} err="failed to get container status \"277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4\": rpc error: code = NotFound desc = could not find container \"277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4\": container with ID starting with 277e927ebe951357221f54a7a124e239b913f8dabcba3f75c5e4f62eafca1ff4 not found: ID does not exist" Jul 24 00:45:30.987577 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.987505 2576 scope.go:117] "RemoveContainer" containerID="3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4" Jul 24 00:45:30.987759 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:45:30.987724 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4\": container with ID starting with 3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4 not found: ID does not exist" containerID="3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4" Jul 24 00:45:30.987801 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.987763 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4"} err="failed to get container status \"3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4\": rpc error: code = NotFound desc = could not find container \"3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4\": container with ID starting with 3456762ffdc9c058096c5ae5ed705f60946a3c18cd46386b710b7806a2c622b4 not found: ID does not exist" Jul 24 00:45:30.988355 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.988337 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x"] Jul 24 00:45:30.993301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:30.993277 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-587b8d23-openshift-default-5955d9f8b4-wht7x"] Jul 24 00:45:32.002020 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:32.001987 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" path="/var/lib/kubelet/pods/ec3d947b-f999-40ef-b087-c53c673e5ce0/volumes" Jul 24 00:45:44.397320 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397231 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt"] Jul 24 00:45:44.397723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397578 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3176b495-c5ee-4a88-b57a-db7206c50c27" containerName="cleanup" Jul 24 00:45:44.397723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397590 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3176b495-c5ee-4a88-b57a-db7206c50c27" containerName="cleanup" Jul 24 00:45:44.397723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397602 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4475610-1280-485a-8c3f-ea7324fb9d9a" containerName="revoke-keys" Jul 24 00:45:44.397723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397607 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4475610-1280-485a-8c3f-ea7324fb9d9a" containerName="revoke-keys" Jul 24 00:45:44.397723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397617 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3176b495-c5ee-4a88-b57a-db7206c50c27" containerName="cleanup" Jul 24 00:45:44.397723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397623 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3176b495-c5ee-4a88-b57a-db7206c50c27" containerName="cleanup" Jul 24 00:45:44.397723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397632 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd13eda-c959-4950-b88d-c70b141fb37f" containerName="cleanup" Jul 24 00:45:44.397723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397638 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd13eda-c959-4950-b88d-c70b141fb37f" containerName="cleanup" Jul 24 00:45:44.397723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397656 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" Jul 24 00:45:44.397723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397664 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" Jul 24 00:45:44.397723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397682 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" Jul 24 00:45:44.397723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397690 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" Jul 24 00:45:44.398289 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397765 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd13eda-c959-4950-b88d-c70b141fb37f" containerName="cleanup" Jul 24 00:45:44.398289 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397775 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3176b495-c5ee-4a88-b57a-db7206c50c27" containerName="cleanup" Jul 24 00:45:44.398289 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397785 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3176b495-c5ee-4a88-b57a-db7206c50c27" containerName="cleanup" Jul 24 00:45:44.398289 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397790 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4475610-1280-485a-8c3f-ea7324fb9d9a" containerName="revoke-keys" Jul 24 00:45:44.398289 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397797 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" Jul 24 00:45:44.398289 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.397802 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec3d947b-f999-40ef-b087-c53c673e5ce0" containerName="istio-proxy" Jul 24 00:45:44.402550 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.402526 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.405034 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.405010 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c4e06e57-openshift-default-dockercfg-4b7mv\"" Jul 24 00:45:44.414508 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.414480 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt"] Jul 24 00:45:44.587242 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.587205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-workload-certs\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.587425 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.587263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-podinfo\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.587425 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.587359 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-token\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.587521 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.587418 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-envoy\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.587521 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.587451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5g7s\" (UniqueName: \"kubernetes.io/projected/a1654912-2da1-4ba5-94f3-cfaa286ffc80-kube-api-access-p5g7s\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.587521 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.587490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-data\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.587521 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.587512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istiod-ca-cert\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.587665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.587531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-workload-socket\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.587665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.587576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-credential-socket\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.688790 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.688687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-data\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.688790 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.688730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istiod-ca-cert\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.688790 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.688750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-workload-socket\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.688790 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.688779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-credential-socket\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.689160 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.688815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-workload-certs\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.689160 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.688874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-podinfo\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.689160 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.688920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-token\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.689160 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.688973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-envoy\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.689160 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.688998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5g7s\" (UniqueName: \"kubernetes.io/projected/a1654912-2da1-4ba5-94f3-cfaa286ffc80-kube-api-access-p5g7s\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.689414 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.689192 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-data\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.689414 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.689281 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-workload-socket\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.689532 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.689420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-workload-certs\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.689532 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.689523 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-credential-socket\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.689689 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.689666 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istiod-ca-cert\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.691451 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.691419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-envoy\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.691618 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.691600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-podinfo\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.699909 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.699881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-token\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.700297 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.700273 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5g7s\" (UniqueName: \"kubernetes.io/projected/a1654912-2da1-4ba5-94f3-cfaa286ffc80-kube-api-access-p5g7s\") pod \"e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.714023 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.714000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:44.854672 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.854644 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt"] Jul 24 00:45:44.856704 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:45:44.856679 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1654912_2da1_4ba5_94f3_cfaa286ffc80.slice/crio-7a7d400c363f2fe0897be0556dae656a3bac37b270d202de88e0d1bf609232c0 WatchSource:0}: Error finding container 7a7d400c363f2fe0897be0556dae656a3bac37b270d202de88e0d1bf609232c0: Status 404 returned error can't find the container with id 7a7d400c363f2fe0897be0556dae656a3bac37b270d202de88e0d1bf609232c0 Jul 24 00:45:44.858812 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.858783 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:45:44.858886 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.858851 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:45:44.858886 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:44.858880 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:45:45.043621 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:45.043586 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" event={"ID":"a1654912-2da1-4ba5-94f3-cfaa286ffc80","Type":"ContainerStarted","Data":"8f0f84b41c12b81e07df647644806202b2f9b0bd40b687f422b0c8d81870c454"} Jul 24 00:45:45.043798 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:45.043647 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" event={"ID":"a1654912-2da1-4ba5-94f3-cfaa286ffc80","Type":"ContainerStarted","Data":"7a7d400c363f2fe0897be0556dae656a3bac37b270d202de88e0d1bf609232c0"} Jul 24 00:45:45.066237 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:45.066181 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" podStartSLOduration=1.06616201 podStartE2EDuration="1.06616201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:45:45.064438901 +0000 UTC m=+1921.637535705" watchObservedRunningTime="2026-07-24 00:45:45.06616201 +0000 UTC m=+1921.639258812" Jul 24 00:45:45.715188 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:45.715146 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:45.720615 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:45.720584 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:46.049716 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:46.049691 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:46.050719 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:46.050694 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:49.587443 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:49.587406 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:45:58.117920 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:58.117886 2576 generic.go:356] "Generic (PLEG): container finished" podID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerID="8f0f84b41c12b81e07df647644806202b2f9b0bd40b687f422b0c8d81870c454" exitCode=0 Jul 24 00:45:58.118371 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:58.117975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" event={"ID":"a1654912-2da1-4ba5-94f3-cfaa286ffc80","Type":"ContainerDied","Data":"8f0f84b41c12b81e07df647644806202b2f9b0bd40b687f422b0c8d81870c454"} Jul 24 00:45:58.118415 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:58.118384 2576 scope.go:117] "RemoveContainer" containerID="8f0f84b41c12b81e07df647644806202b2f9b0bd40b687f422b0c8d81870c454" Jul 24 00:45:58.119416 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:58.119383 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:45:58.119521 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:58.119455 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:45:58.119521 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:58.119484 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:45:59.125432 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:59.125397 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" event={"ID":"a1654912-2da1-4ba5-94f3-cfaa286ffc80","Type":"ContainerStarted","Data":"7cc96a69d2dedda83e2ca2b84046e17660b78eb1c5a52ae4b2a0e145a0e1089a"} Jul 24 00:45:59.714891 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:59.714851 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:45:59.715941 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:59.715909 2576 patch_prober.go:28] interesting pod/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 24 00:45:59.716085 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:45:59.715988 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" podUID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 24 00:46:00.715081 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:00.715035 2576 patch_prober.go:28] interesting pod/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 24 00:46:00.715588 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:00.715112 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" podUID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 24 00:46:01.050588 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:01.050540 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:46:01.714760 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:01.714714 2576 patch_prober.go:28] interesting pod/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 24 00:46:01.714977 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:01.714776 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" podUID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 24 00:46:02.714527 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:02.714490 2576 patch_prober.go:28] interesting pod/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 24 00:46:02.714936 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:02.714576 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" podUID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 24 00:46:03.718454 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:03.718420 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:46:03.719520 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:03.719497 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:46:03.981551 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:03.981459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:46:21.297839 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:21.297804 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:46:26.201033 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:26.200995 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc"] Jul 24 00:46:26.204441 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:26.204413 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc" Jul 24 00:46:26.206822 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:26.206802 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 00:46:26.206969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:26.206902 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\"" Jul 24 00:46:26.211750 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:26.211716 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc"] Jul 24 00:46:26.305249 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:26.305218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d673d1-4fe5-457e-afcd-50c335205e56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc\" (UID: \"23d673d1-4fe5-457e-afcd-50c335205e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc" Jul 24 00:46:26.406006 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:26.405941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d673d1-4fe5-457e-afcd-50c335205e56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc\" (UID: \"23d673d1-4fe5-457e-afcd-50c335205e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc" Jul 24 00:46:26.406565 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:26.406543 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d673d1-4fe5-457e-afcd-50c335205e56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc\" (UID: \"23d673d1-4fe5-457e-afcd-50c335205e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc" Jul 24 00:46:26.515771 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:26.515667 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc" Jul 24 00:46:26.662050 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:26.662020 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc"] Jul 24 00:46:26.663489 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:46:26.663458 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d673d1_4fe5_457e_afcd_50c335205e56.slice/crio-898a1cddbe025892464260cc2ac0a957f6ab6295937fd36ef480748fe30d5a91 WatchSource:0}: Error finding container 898a1cddbe025892464260cc2ac0a957f6ab6295937fd36ef480748fe30d5a91: Status 404 returned error can't find the container with id 898a1cddbe025892464260cc2ac0a957f6ab6295937fd36ef480748fe30d5a91 Jul 24 00:46:27.294016 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:27.293982 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc" event={"ID":"23d673d1-4fe5-457e-afcd-50c335205e56","Type":"ContainerStarted","Data":"56e301962559903aaf0c390783a278a1138059876ea3888222d6967592b7ef51"} Jul 24 00:46:27.294404 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:27.294049 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc" event={"ID":"23d673d1-4fe5-457e-afcd-50c335205e56","Type":"ContainerStarted","Data":"898a1cddbe025892464260cc2ac0a957f6ab6295937fd36ef480748fe30d5a91"} Jul 24 00:46:27.310624 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:27.310573 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc" podStartSLOduration=1.310555996 podStartE2EDuration="1.310555996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:46:27.309026221 +0000 UTC m=+1963.882123021" watchObservedRunningTime="2026-07-24 00:46:27.310555996 +0000 UTC m=+1963.883652822" Jul 24 00:46:28.301753 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:28.301713 2576 generic.go:356] "Generic (PLEG): container finished" podID="23d673d1-4fe5-457e-afcd-50c335205e56" containerID="56e301962559903aaf0c390783a278a1138059876ea3888222d6967592b7ef51" exitCode=0 Jul 24 00:46:28.302314 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:28.301803 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc" event={"ID":"23d673d1-4fe5-457e-afcd-50c335205e56","Type":"ContainerDied","Data":"56e301962559903aaf0c390783a278a1138059876ea3888222d6967592b7ef51"} Jul 24 00:46:29.450126 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:29.450098 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc" Jul 24 00:46:29.533156 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:29.533121 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d673d1-4fe5-457e-afcd-50c335205e56-maas-api-service-ca\") pod \"23d673d1-4fe5-457e-afcd-50c335205e56\" (UID: \"23d673d1-4fe5-457e-afcd-50c335205e56\") " Jul 24 00:46:29.533523 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:29.533496 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d673d1-4fe5-457e-afcd-50c335205e56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23d673d1-4fe5-457e-afcd-50c335205e56" (UID: "23d673d1-4fe5-457e-afcd-50c335205e56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:46:29.634071 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:29.633977 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d673d1-4fe5-457e-afcd-50c335205e56-maas-api-service-ca\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:46:30.315030 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:30.315002 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc" Jul 24 00:46:30.315214 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:30.315030 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc" event={"ID":"23d673d1-4fe5-457e-afcd-50c335205e56","Type":"ContainerDied","Data":"898a1cddbe025892464260cc2ac0a957f6ab6295937fd36ef480748fe30d5a91"} Jul 24 00:46:30.315214 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:30.315076 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="898a1cddbe025892464260cc2ac0a957f6ab6295937fd36ef480748fe30d5a91" Jul 24 00:46:40.699191 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:40.699093 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:46:46.279443 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:46.279402 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc"] Jul 24 00:46:46.282427 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:46.282397 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4e06e57-thdxc"] Jul 24 00:46:48.002525 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:48.002490 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23d673d1-4fe5-457e-afcd-50c335205e56" path="/var/lib/kubelet/pods/23d673d1-4fe5-457e-afcd-50c335205e56/volumes" Jul 24 00:46:49.190086 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:46:49.190048 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:47:37.510871 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:37.510828 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt"] Jul 24 00:47:37.511527 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:37.511109 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" podUID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerName="istio-proxy" containerID="cri-o://7cc96a69d2dedda83e2ca2b84046e17660b78eb1c5a52ae4b2a0e145a0e1089a" gracePeriod=30 Jul 24 00:47:41.198217 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:41.198179 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:47:42.727296 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.727263 2576 generic.go:356] "Generic (PLEG): container finished" podID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerID="7cc96a69d2dedda83e2ca2b84046e17660b78eb1c5a52ae4b2a0e145a0e1089a" exitCode=0 Jul 24 00:47:42.727726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.727331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" event={"ID":"a1654912-2da1-4ba5-94f3-cfaa286ffc80","Type":"ContainerDied","Data":"7cc96a69d2dedda83e2ca2b84046e17660b78eb1c5a52ae4b2a0e145a0e1089a"} Jul 24 00:47:42.727726 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.727399 2576 scope.go:117] "RemoveContainer" containerID="8f0f84b41c12b81e07df647644806202b2f9b0bd40b687f422b0c8d81870c454" Jul 24 00:47:42.771318 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.771249 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:47:42.909500 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.909465 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istiod-ca-cert\") pod \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " Jul 24 00:47:42.909698 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.909518 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-data\") pod \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " Jul 24 00:47:42.909698 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.909546 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-envoy\") pod \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " Jul 24 00:47:42.910035 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.909776 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5g7s\" (UniqueName: \"kubernetes.io/projected/a1654912-2da1-4ba5-94f3-cfaa286ffc80-kube-api-access-p5g7s\") pod \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " Jul 24 00:47:42.910182 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.910081 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-podinfo\") pod \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " Jul 24 00:47:42.910182 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.910123 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-workload-socket\") pod \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " Jul 24 00:47:42.910314 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.910191 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-token\") pod \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " Jul 24 00:47:42.910314 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.910235 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-workload-certs\") pod \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " Jul 24 00:47:42.910314 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.910282 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-credential-socket\") pod \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\" (UID: \"a1654912-2da1-4ba5-94f3-cfaa286ffc80\") " Jul 24 00:47:42.910469 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.910336 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1654912-2da1-4ba5-94f3-cfaa286ffc80" (UID: "a1654912-2da1-4ba5-94f3-cfaa286ffc80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:47:42.911008 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.910612 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1654912-2da1-4ba5-94f3-cfaa286ffc80" (UID: "a1654912-2da1-4ba5-94f3-cfaa286ffc80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:47:42.911008 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.910976 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1654912-2da1-4ba5-94f3-cfaa286ffc80" (UID: "a1654912-2da1-4ba5-94f3-cfaa286ffc80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:47:42.911292 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.911264 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1654912-2da1-4ba5-94f3-cfaa286ffc80" (UID: "a1654912-2da1-4ba5-94f3-cfaa286ffc80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:47:42.911602 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.911581 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1654912-2da1-4ba5-94f3-cfaa286ffc80" (UID: "a1654912-2da1-4ba5-94f3-cfaa286ffc80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:47:42.911990 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.911966 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:47:42.912072 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.911998 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:47:42.912072 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.912014 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:47:42.912072 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.912029 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:47:42.912760 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.912727 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1654912-2da1-4ba5-94f3-cfaa286ffc80" (UID: "a1654912-2da1-4ba5-94f3-cfaa286ffc80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:47:42.912760 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.912739 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1654912-2da1-4ba5-94f3-cfaa286ffc80-kube-api-access-p5g7s" (OuterVolumeSpecName: "kube-api-access-p5g7s") pod "a1654912-2da1-4ba5-94f3-cfaa286ffc80" (UID: "a1654912-2da1-4ba5-94f3-cfaa286ffc80"). InnerVolumeSpecName "kube-api-access-p5g7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:47:42.919346 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.919297 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1654912-2da1-4ba5-94f3-cfaa286ffc80" (UID: "a1654912-2da1-4ba5-94f3-cfaa286ffc80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 00:47:42.920889 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:42.920865 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1654912-2da1-4ba5-94f3-cfaa286ffc80" (UID: "a1654912-2da1-4ba5-94f3-cfaa286ffc80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:47:43.012659 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:43.012621 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:47:43.012659 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:43.012656 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5g7s\" (UniqueName: \"kubernetes.io/projected/a1654912-2da1-4ba5-94f3-cfaa286ffc80-kube-api-access-p5g7s\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:47:43.012659 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:43.012667 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:47:43.012885 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:43.012676 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1654912-2da1-4ba5-94f3-cfaa286ffc80-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:47:43.012885 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:43.012685 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1654912-2da1-4ba5-94f3-cfaa286ffc80-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:47:43.734519 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:43.734486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" event={"ID":"a1654912-2da1-4ba5-94f3-cfaa286ffc80","Type":"ContainerDied","Data":"7a7d400c363f2fe0897be0556dae656a3bac37b270d202de88e0d1bf609232c0"} Jul 24 00:47:43.734519 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:43.734521 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt" Jul 24 00:47:43.735083 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:43.734545 2576 scope.go:117] "RemoveContainer" containerID="7cc96a69d2dedda83e2ca2b84046e17660b78eb1c5a52ae4b2a0e145a0e1089a" Jul 24 00:47:43.758286 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:43.758248 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt"] Jul 24 00:47:43.761909 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:43.761873 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c4e06e57-openshift-default-6f4cc5f8fb-4kgtt"] Jul 24 00:47:44.002817 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:44.002737 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" path="/var/lib/kubelet/pods/a1654912-2da1-4ba5-94f3-cfaa286ffc80/volumes" Jul 24 00:47:44.151937 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:44.151897 2576 scope.go:117] "RemoveContainer" containerID="4cd766af51596908b090df937a638228cda7d45927d7b5713e6a03814755ac34" Jul 24 00:47:44.160865 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:44.160843 2576 scope.go:117] "RemoveContainer" containerID="9fd2d4a7bb8947d719de260e1433b542549357a04efe0c8dcbb9d3e2017977a8" Jul 24 00:47:51.090766 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:47:51.090726 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:48:33.135863 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.135829 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv"] Jul 24 00:48:33.136371 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.136191 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d673d1-4fe5-457e-afcd-50c335205e56" containerName="revoke-keys" Jul 24 00:48:33.136371 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.136203 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d673d1-4fe5-457e-afcd-50c335205e56" containerName="revoke-keys" Jul 24 00:48:33.136371 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.136225 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerName="istio-proxy" Jul 24 00:48:33.136371 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.136230 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerName="istio-proxy" Jul 24 00:48:33.136371 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.136239 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerName="istio-proxy" Jul 24 00:48:33.136371 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.136244 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerName="istio-proxy" Jul 24 00:48:33.136371 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.136300 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d673d1-4fe5-457e-afcd-50c335205e56" containerName="revoke-keys" Jul 24 00:48:33.136371 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.136307 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerName="istio-proxy" Jul 24 00:48:33.139479 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.139462 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.142076 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.142054 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ac89bc-openshift-default-dockercfg-vcr2m\"" Jul 24 00:48:33.150264 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.150227 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv"] Jul 24 00:48:33.278700 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.278650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-podinfo\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.278905 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.278715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-data\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.278905 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.278762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-credential-socket\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.278905 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.278783 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-workload-socket\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.278905 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.278804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-token\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.278905 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.278820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gcl\" (UniqueName: \"kubernetes.io/projected/77aff1e0-41e3-4853-851f-2c7b2566035c-kube-api-access-f4gcl\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.279126 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.278900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aff1e0-41e3-4853-851f-2c7b2566035c-istiod-ca-cert\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.279126 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.279015 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-workload-certs\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.279126 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.279032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-envoy\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.379917 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.379881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-workload-certs\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.379917 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.379919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-envoy\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.380239 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.379938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-podinfo\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.380239 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.379990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-data\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.380239 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.380040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-credential-socket\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.380239 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.380064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-workload-socket\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.380239 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.380085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-token\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.380239 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.380108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gcl\" (UniqueName: \"kubernetes.io/projected/77aff1e0-41e3-4853-851f-2c7b2566035c-kube-api-access-f4gcl\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.380239 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.380158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aff1e0-41e3-4853-851f-2c7b2566035c-istiod-ca-cert\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.380596 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.380357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-workload-certs\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.380596 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.380429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-credential-socket\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.380596 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.380450 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-workload-socket\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.380759 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.380719 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-data\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.381131 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.381109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aff1e0-41e3-4853-851f-2c7b2566035c-istiod-ca-cert\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.382722 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.382695 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-envoy\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.382882 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.382866 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-podinfo\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.391474 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.391408 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-token\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.391753 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.391735 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gcl\" (UniqueName: \"kubernetes.io/projected/77aff1e0-41e3-4853-851f-2c7b2566035c-kube-api-access-f4gcl\") pod \"e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.451629 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.451586 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:33.615231 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.615197 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv"] Jul 24 00:48:33.616678 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:48:33.616640 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77aff1e0_41e3_4853_851f_2c7b2566035c.slice/crio-2500b22e4765f592744f8ec3cb3b65bcb1e1fd4dfcc1a42eec660d3486503027 WatchSource:0}: Error finding container 2500b22e4765f592744f8ec3cb3b65bcb1e1fd4dfcc1a42eec660d3486503027: Status 404 returned error can't find the container with id 2500b22e4765f592744f8ec3cb3b65bcb1e1fd4dfcc1a42eec660d3486503027 Jul 24 00:48:33.618755 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.618725 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:48:33.618839 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.618798 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:48:33.618839 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:33.618829 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:48:34.016040 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:34.015941 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" event={"ID":"77aff1e0-41e3-4853-851f-2c7b2566035c","Type":"ContainerStarted","Data":"fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72"} Jul 24 00:48:34.016040 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:34.016008 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" event={"ID":"77aff1e0-41e3-4853-851f-2c7b2566035c","Type":"ContainerStarted","Data":"2500b22e4765f592744f8ec3cb3b65bcb1e1fd4dfcc1a42eec660d3486503027"} Jul 24 00:48:34.036546 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:34.036495 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" podStartSLOduration=1.036478238 podStartE2EDuration="1.036478238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:48:34.03421418 +0000 UTC m=+2090.607310999" watchObservedRunningTime="2026-07-24 00:48:34.036478238 +0000 UTC m=+2090.609575041" Jul 24 00:48:34.452409 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:34.452376 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:34.457384 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:34.457358 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:35.022460 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:35.022415 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:35.023551 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:35.023533 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:41.221853 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.221820 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj"] Jul 24 00:48:41.222287 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.222273 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1654912-2da1-4ba5-94f3-cfaa286ffc80" containerName="istio-proxy" Jul 24 00:48:41.225512 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.225491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.228066 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.228043 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e7050a-openshift-default-dockercfg-g2cfq\"" Jul 24 00:48:41.233771 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.233721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-envoy\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.233847 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.233811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-token\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.233847 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.233837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkvkp\" (UniqueName: \"kubernetes.io/projected/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-kube-api-access-rkvkp\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.233994 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.233967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istiod-ca-cert\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.234053 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.234026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-workload-socket\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.234119 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.234093 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-credential-socket\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.234183 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.234141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-podinfo\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.234183 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.234172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-workload-certs\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.234293 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.234213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-data\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.238050 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.238029 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj"] Jul 24 00:48:41.334756 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.334719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-token\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.334756 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.334758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkvkp\" (UniqueName: \"kubernetes.io/projected/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-kube-api-access-rkvkp\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.335064 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.334785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istiod-ca-cert\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.335128 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.335067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-workload-socket\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.335202 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.335157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-credential-socket\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.335261 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.335225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-podinfo\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.335261 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.335254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-workload-certs\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.335374 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.335301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-data\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.335374 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.335363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-envoy\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.335500 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.335476 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-workload-socket\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.335791 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.335732 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istiod-ca-cert\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.335791 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.335621 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-workload-certs\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.335929 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.335804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-credential-socket\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.335929 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.335825 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-data\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.337723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.337701 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-envoy\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.337835 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.337810 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-podinfo\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.344014 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.343941 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-token\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.344300 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.344273 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkvkp\" (UniqueName: \"kubernetes.io/projected/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-kube-api-access-rkvkp\") pod \"e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.538568 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.538473 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:41.683332 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.683288 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj"] Jul 24 00:48:41.684103 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:48:41.684076 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181a1da7_3181_4aa7_8b44_98e0ffa9eac4.slice/crio-3fd12c4d6be15501cdbb4ec1cf63eb7cb960cca96034e0826f024e292729ba99 WatchSource:0}: Error finding container 3fd12c4d6be15501cdbb4ec1cf63eb7cb960cca96034e0826f024e292729ba99: Status 404 returned error can't find the container with id 3fd12c4d6be15501cdbb4ec1cf63eb7cb960cca96034e0826f024e292729ba99 Jul 24 00:48:41.686161 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.686129 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:48:41.686272 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.686195 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:48:41.686272 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.686227 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:48:41.697080 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:41.697055 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:48:42.067163 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:42.067126 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" event={"ID":"181a1da7-3181-4aa7-8b44-98e0ffa9eac4","Type":"ContainerStarted","Data":"d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2"} Jul 24 00:48:42.067363 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:42.067204 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" event={"ID":"181a1da7-3181-4aa7-8b44-98e0ffa9eac4","Type":"ContainerStarted","Data":"3fd12c4d6be15501cdbb4ec1cf63eb7cb960cca96034e0826f024e292729ba99"} Jul 24 00:48:42.089677 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:42.089620 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" podStartSLOduration=1.089602768 podStartE2EDuration="1.089602768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:48:42.08677218 +0000 UTC m=+2098.659868980" watchObservedRunningTime="2026-07-24 00:48:42.089602768 +0000 UTC m=+2098.662699572" Jul 24 00:48:42.539691 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:42.539660 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:42.553672 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:42.553616 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:43.072881 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:43.072851 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:43.074056 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:43.074035 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:44.098456 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:44.098424 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:48:44.103977 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:44.103935 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:48:44.190695 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:44.190661 2576 scope.go:117] "RemoveContainer" containerID="83b1a03cecff3623e9d97c310d9af2a99bedd3880b004b67b59c923447bd3b71" Jul 24 00:48:50.023538 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:50.023452 2576 patch_prober.go:28] interesting pod/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 24 00:48:50.023538 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:50.023514 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 24 00:48:50.119849 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:50.119816 2576 generic.go:356] "Generic (PLEG): container finished" podID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerID="fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72" exitCode=0 Jul 24 00:48:50.120047 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:50.119889 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" event={"ID":"77aff1e0-41e3-4853-851f-2c7b2566035c","Type":"ContainerDied","Data":"fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72"} Jul 24 00:48:50.120312 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:50.120293 2576 scope.go:117] "RemoveContainer" containerID="fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72" Jul 24 00:48:50.121262 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:50.121234 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:48:50.121361 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:50.121298 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:48:50.121361 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:50.121331 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:48:51.127196 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:51.127152 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" event={"ID":"77aff1e0-41e3-4853-851f-2c7b2566035c","Type":"ContainerStarted","Data":"3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf"} Jul 24 00:48:51.452275 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:51.452164 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:51.452480 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:51.452344 2576 patch_prober.go:28] interesting pod/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 24 00:48:51.452480 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:51.452410 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 24 00:48:52.453136 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:52.453025 2576 patch_prober.go:28] interesting pod/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 24 00:48:52.453136 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:52.453098 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 24 00:48:53.452940 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:53.452474 2576 patch_prober.go:28] interesting pod/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 24 00:48:53.452940 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:53.452539 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 24 00:48:54.093135 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:54.093095 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:48:54.453113 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:54.453012 2576 patch_prober.go:28] interesting pod/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 24 00:48:54.453374 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:54.453340 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 24 00:48:55.456998 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:55.456938 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:55.457382 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:55.457246 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:55.458092 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:55.458062 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:48:56.162367 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:56.162332 2576 generic.go:356] "Generic (PLEG): container finished" podID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerID="d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2" exitCode=0 Jul 24 00:48:56.162619 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:56.162403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" event={"ID":"181a1da7-3181-4aa7-8b44-98e0ffa9eac4","Type":"ContainerDied","Data":"d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2"} Jul 24 00:48:56.162821 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:56.162805 2576 scope.go:117] "RemoveContainer" containerID="d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2" Jul 24 00:48:56.170096 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:56.164257 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:48:56.170096 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:56.164666 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:48:56.170096 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:56.164762 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:48:57.172306 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:57.171499 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" event={"ID":"181a1da7-3181-4aa7-8b44-98e0ffa9eac4","Type":"ContainerStarted","Data":"6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d"} Jul 24 00:48:57.539866 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:57.539833 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:57.541110 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:57.541076 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 24 00:48:57.541328 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:57.541300 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" podUID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 24 00:48:58.073548 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:58.073498 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:48:58.547973 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:58.544372 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 24 00:48:58.547973 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:58.544441 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" podUID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 24 00:48:59.539854 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:59.539807 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 24 00:48:59.540097 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:48:59.539878 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" podUID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 24 00:49:00.540098 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:00.540052 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 24 00:49:00.540607 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:00.540122 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" podUID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 24 00:49:01.543456 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:01.543421 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:49:01.544627 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:01.544605 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:49:03.595449 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:03.595415 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:49:21.177669 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.177635 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2"] Jul 24 00:49:21.179911 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.179891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.183473 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.183448 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e7050a\"/\"test-model-e7050a-kserve-self-signed-certs\"" Jul 24 00:49:21.183612 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.183448 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e7050a\"/\"default-dockercfg-8c9ss\"" Jul 24 00:49:21.183612 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.183449 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e7050a\"/\"kube-root-ca.crt\"" Jul 24 00:49:21.184215 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.184195 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e7050a\"/\"openshift-service-ca.crt\"" Jul 24 00:49:21.191495 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.191474 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2"] Jul 24 00:49:21.314717 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.314687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-home\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.314897 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.314730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-model-cache\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.314897 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.314789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpwmz\" (UniqueName: \"kubernetes.io/projected/72b518c9-985f-451b-8a59-eb65a018d8b3-kube-api-access-kpwmz\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.314897 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.314816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b518c9-985f-451b-8a59-eb65a018d8b3-tls-certs\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.314897 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.314877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-dshm\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.315067 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.314930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-tmp-dir\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.415733 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.415695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpwmz\" (UniqueName: \"kubernetes.io/projected/72b518c9-985f-451b-8a59-eb65a018d8b3-kube-api-access-kpwmz\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.415914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.415748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b518c9-985f-451b-8a59-eb65a018d8b3-tls-certs\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.415914 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.415863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-dshm\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.416052 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.415939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-tmp-dir\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.416052 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.416013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-home\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.416139 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.416071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-model-cache\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.416386 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.416363 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-tmp-dir\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.416386 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.416376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-model-cache\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.416529 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.416414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-home\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.417962 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.417926 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-dshm\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.418228 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.418211 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b518c9-985f-451b-8a59-eb65a018d8b3-tls-certs\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.423618 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.423593 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpwmz\" (UniqueName: \"kubernetes.io/projected/72b518c9-985f-451b-8a59-eb65a018d8b3-kube-api-access-kpwmz\") pod \"test-model-e7050a-kserve-5cfb6f9c69-kr8c2\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.490107 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.490006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:21.635884 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:21.635854 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2"] Jul 24 00:49:21.637444 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:49:21.637417 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72b518c9_985f_451b_8a59_eb65a018d8b3.slice/crio-ad234be91902a554726f90c4d1fab97d720ad00cebbf4671f64d79051b4335d2 WatchSource:0}: Error finding container ad234be91902a554726f90c4d1fab97d720ad00cebbf4671f64d79051b4335d2: Status 404 returned error can't find the container with id ad234be91902a554726f90c4d1fab97d720ad00cebbf4671f64d79051b4335d2 Jul 24 00:49:22.317216 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:22.317176 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" event={"ID":"72b518c9-985f-451b-8a59-eb65a018d8b3","Type":"ContainerStarted","Data":"4bcf086f90d71d3d8ea07e2ea83e7ca09bfa7854468ab1fe98829276dff210d8"} Jul 24 00:49:22.317614 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:22.317227 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" event={"ID":"72b518c9-985f-451b-8a59-eb65a018d8b3","Type":"ContainerStarted","Data":"ad234be91902a554726f90c4d1fab97d720ad00cebbf4671f64d79051b4335d2"} Jul 24 00:49:22.336249 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:22.336194 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" podStartSLOduration=1.336178541 podStartE2EDuration="1.336178541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:49:22.333219091 +0000 UTC m=+2138.906315919" watchObservedRunningTime="2026-07-24 00:49:22.336178541 +0000 UTC m=+2138.909275343" Jul 24 00:49:25.890866 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:25.890827 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:49:31.490532 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:31.490495 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:31.490532 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:31.490543 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:31.503323 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:31.503290 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:32.386968 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:32.386924 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:49:38.384506 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:38.384404 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:49:44.219993 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:49:44.219936 2576 scope.go:117] "RemoveContainer" containerID="8f7d310875881ff2e55b80d7d22d9447afa1db453a9b9c725db7916b96d46475" Jul 24 00:50:04.436368 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:04.436331 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2"] Jul 24 00:50:08.680975 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:08.680926 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:50:19.455968 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.455893 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" podUID="72b518c9-985f-451b-8a59-eb65a018d8b3" containerName="main" containerID="cri-o://4bcf086f90d71d3d8ea07e2ea83e7ca09bfa7854468ab1fe98829276dff210d8" gracePeriod=45 Jul 24 00:50:19.653690 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.653651 2576 generic.go:356] "Generic (PLEG): container finished" podID="72b518c9-985f-451b-8a59-eb65a018d8b3" containerID="4bcf086f90d71d3d8ea07e2ea83e7ca09bfa7854468ab1fe98829276dff210d8" exitCode=0 Jul 24 00:50:19.653840 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.653720 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" event={"ID":"72b518c9-985f-451b-8a59-eb65a018d8b3","Type":"ContainerDied","Data":"4bcf086f90d71d3d8ea07e2ea83e7ca09bfa7854468ab1fe98829276dff210d8"} Jul 24 00:50:19.712102 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.712030 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:50:19.753532 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.753492 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-model-cache\") pod \"72b518c9-985f-451b-8a59-eb65a018d8b3\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " Jul 24 00:50:19.753739 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.753588 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-tmp-dir\") pod \"72b518c9-985f-451b-8a59-eb65a018d8b3\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " Jul 24 00:50:19.753739 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.753616 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-home\") pod \"72b518c9-985f-451b-8a59-eb65a018d8b3\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " Jul 24 00:50:19.753739 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.753635 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-dshm\") pod \"72b518c9-985f-451b-8a59-eb65a018d8b3\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " Jul 24 00:50:19.753739 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.753717 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b518c9-985f-451b-8a59-eb65a018d8b3-tls-certs\") pod \"72b518c9-985f-451b-8a59-eb65a018d8b3\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " Jul 24 00:50:19.753962 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.753744 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpwmz\" (UniqueName: \"kubernetes.io/projected/72b518c9-985f-451b-8a59-eb65a018d8b3-kube-api-access-kpwmz\") pod \"72b518c9-985f-451b-8a59-eb65a018d8b3\" (UID: \"72b518c9-985f-451b-8a59-eb65a018d8b3\") " Jul 24 00:50:19.753962 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.753769 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "72b518c9-985f-451b-8a59-eb65a018d8b3" (UID: "72b518c9-985f-451b-8a59-eb65a018d8b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:19.753962 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.753881 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72b518c9-985f-451b-8a59-eb65a018d8b3" (UID: "72b518c9-985f-451b-8a59-eb65a018d8b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:19.753962 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.753890 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-home" (OuterVolumeSpecName: "home") pod "72b518c9-985f-451b-8a59-eb65a018d8b3" (UID: "72b518c9-985f-451b-8a59-eb65a018d8b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:19.754130 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.754058 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-tmp-dir\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:19.754130 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.754074 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-home\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:19.754130 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.754087 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-model-cache\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:19.756040 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.756014 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b518c9-985f-451b-8a59-eb65a018d8b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72b518c9-985f-451b-8a59-eb65a018d8b3" (UID: "72b518c9-985f-451b-8a59-eb65a018d8b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:50:19.756192 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.756172 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b518c9-985f-451b-8a59-eb65a018d8b3-kube-api-access-kpwmz" (OuterVolumeSpecName: "kube-api-access-kpwmz") pod "72b518c9-985f-451b-8a59-eb65a018d8b3" (UID: "72b518c9-985f-451b-8a59-eb65a018d8b3"). InnerVolumeSpecName "kube-api-access-kpwmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:50:19.756255 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.756197 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-dshm" (OuterVolumeSpecName: "dshm") pod "72b518c9-985f-451b-8a59-eb65a018d8b3" (UID: "72b518c9-985f-451b-8a59-eb65a018d8b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:19.855213 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.855154 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b518c9-985f-451b-8a59-eb65a018d8b3-dshm\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:19.855213 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.855198 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b518c9-985f-451b-8a59-eb65a018d8b3-tls-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:19.855213 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:19.855216 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpwmz\" (UniqueName: \"kubernetes.io/projected/72b518c9-985f-451b-8a59-eb65a018d8b3-kube-api-access-kpwmz\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:20.660388 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:20.660359 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" Jul 24 00:50:20.660816 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:20.660353 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2" event={"ID":"72b518c9-985f-451b-8a59-eb65a018d8b3","Type":"ContainerDied","Data":"ad234be91902a554726f90c4d1fab97d720ad00cebbf4671f64d79051b4335d2"} Jul 24 00:50:20.660816 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:20.660504 2576 scope.go:117] "RemoveContainer" containerID="4bcf086f90d71d3d8ea07e2ea83e7ca09bfa7854468ab1fe98829276dff210d8" Jul 24 00:50:20.677812 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:20.677781 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2"] Jul 24 00:50:20.681590 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:20.681567 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e7050a/test-model-e7050a-kserve-5cfb6f9c69-kr8c2"] Jul 24 00:50:22.002347 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:22.002313 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b518c9-985f-451b-8a59-eb65a018d8b3" path="/var/lib/kubelet/pods/72b518c9-985f-451b-8a59-eb65a018d8b3/volumes" Jul 24 00:50:24.582533 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:24.582492 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:50:25.973192 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:25.973155 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv"] Jul 24 00:50:25.973553 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:25.973433 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerName="istio-proxy" containerID="cri-o://3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf" gracePeriod=30 Jul 24 00:50:31.233771 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.233746 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:50:31.361663 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.361624 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-data\") pod \"77aff1e0-41e3-4853-851f-2c7b2566035c\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " Jul 24 00:50:31.361829 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.361732 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-workload-certs\") pod \"77aff1e0-41e3-4853-851f-2c7b2566035c\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " Jul 24 00:50:31.361829 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.361776 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-envoy\") pod \"77aff1e0-41e3-4853-851f-2c7b2566035c\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " Jul 24 00:50:31.361829 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.361804 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-workload-socket\") pod \"77aff1e0-41e3-4853-851f-2c7b2566035c\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " Jul 24 00:50:31.362042 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.361838 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-podinfo\") pod \"77aff1e0-41e3-4853-851f-2c7b2566035c\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " Jul 24 00:50:31.362042 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.361865 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4gcl\" (UniqueName: \"kubernetes.io/projected/77aff1e0-41e3-4853-851f-2c7b2566035c-kube-api-access-f4gcl\") pod \"77aff1e0-41e3-4853-851f-2c7b2566035c\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " Jul 24 00:50:31.362042 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.361907 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aff1e0-41e3-4853-851f-2c7b2566035c-istiod-ca-cert\") pod \"77aff1e0-41e3-4853-851f-2c7b2566035c\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " Jul 24 00:50:31.362042 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.361985 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-credential-socket\") pod \"77aff1e0-41e3-4853-851f-2c7b2566035c\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " Jul 24 00:50:31.362042 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.362010 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-token\") pod \"77aff1e0-41e3-4853-851f-2c7b2566035c\" (UID: \"77aff1e0-41e3-4853-851f-2c7b2566035c\") " Jul 24 00:50:31.362295 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.362062 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77aff1e0-41e3-4853-851f-2c7b2566035c" (UID: "77aff1e0-41e3-4853-851f-2c7b2566035c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:31.362295 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.362210 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77aff1e0-41e3-4853-851f-2c7b2566035c" (UID: "77aff1e0-41e3-4853-851f-2c7b2566035c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:31.362385 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.362296 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77aff1e0-41e3-4853-851f-2c7b2566035c" (UID: "77aff1e0-41e3-4853-851f-2c7b2566035c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:31.362431 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.362392 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:31.362431 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.362409 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:31.362431 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.362422 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:31.362571 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.362400 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77aff1e0-41e3-4853-851f-2c7b2566035c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77aff1e0-41e3-4853-851f-2c7b2566035c" (UID: "77aff1e0-41e3-4853-851f-2c7b2566035c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:50:31.362571 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.362509 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-data" (OuterVolumeSpecName: "istio-data") pod "77aff1e0-41e3-4853-851f-2c7b2566035c" (UID: "77aff1e0-41e3-4853-851f-2c7b2566035c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:31.364210 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.364182 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77aff1e0-41e3-4853-851f-2c7b2566035c" (UID: "77aff1e0-41e3-4853-851f-2c7b2566035c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:31.364329 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.364229 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77aff1e0-41e3-4853-851f-2c7b2566035c-kube-api-access-f4gcl" (OuterVolumeSpecName: "kube-api-access-f4gcl") pod "77aff1e0-41e3-4853-851f-2c7b2566035c" (UID: "77aff1e0-41e3-4853-851f-2c7b2566035c"). InnerVolumeSpecName "kube-api-access-f4gcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:50:31.364329 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.364241 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-token" (OuterVolumeSpecName: "istio-token") pod "77aff1e0-41e3-4853-851f-2c7b2566035c" (UID: "77aff1e0-41e3-4853-851f-2c7b2566035c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:50:31.364329 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.364314 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77aff1e0-41e3-4853-851f-2c7b2566035c" (UID: "77aff1e0-41e3-4853-851f-2c7b2566035c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 00:50:31.463120 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.463083 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aff1e0-41e3-4853-851f-2c7b2566035c-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:31.463120 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.463112 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:31.463120 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.463121 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:31.463120 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.463130 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:31.463415 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.463137 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aff1e0-41e3-4853-851f-2c7b2566035c-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:31.463415 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.463145 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4gcl\" (UniqueName: \"kubernetes.io/projected/77aff1e0-41e3-4853-851f-2c7b2566035c-kube-api-access-f4gcl\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:31.725626 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.725535 2576 generic.go:356] "Generic (PLEG): container finished" podID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerID="3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf" exitCode=0 Jul 24 00:50:31.725626 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.725586 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" Jul 24 00:50:31.725824 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.725617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" event={"ID":"77aff1e0-41e3-4853-851f-2c7b2566035c","Type":"ContainerDied","Data":"3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf"} Jul 24 00:50:31.725824 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.725679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv" event={"ID":"77aff1e0-41e3-4853-851f-2c7b2566035c","Type":"ContainerDied","Data":"2500b22e4765f592744f8ec3cb3b65bcb1e1fd4dfcc1a42eec660d3486503027"} Jul 24 00:50:31.725824 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.725709 2576 scope.go:117] "RemoveContainer" containerID="3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf" Jul 24 00:50:31.739470 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.739449 2576 scope.go:117] "RemoveContainer" containerID="fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72" Jul 24 00:50:31.747878 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.747850 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv"] Jul 24 00:50:31.748438 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.748418 2576 scope.go:117] "RemoveContainer" containerID="3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf" Jul 24 00:50:31.748855 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:50:31.748802 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf\": container with ID starting with 3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf not found: ID does not exist" containerID="3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf" Jul 24 00:50:31.748974 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.748864 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf"} err="failed to get container status \"3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf\": rpc error: code = NotFound desc = could not find container \"3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf\": container with ID starting with 3892ae93f47f4bcfd945d2f3195642c93994c2d70433d04e0e3990be031f26bf not found: ID does not exist" Jul 24 00:50:31.748974 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.748894 2576 scope.go:117] "RemoveContainer" containerID="fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72" Jul 24 00:50:31.749443 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:50:31.749373 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72\": container with ID starting with fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72 not found: ID does not exist" containerID="fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72" Jul 24 00:50:31.749443 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.749407 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72"} err="failed to get container status \"fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72\": rpc error: code = NotFound desc = could not find container \"fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72\": container with ID starting with fa1c6a078ca56bc00300317c60b64fac913926ade528970d4a42dbdf36b31c72 not found: ID does not exist" Jul 24 00:50:31.751624 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:31.751605 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ac89bc-openshift-default-64b7cc4d74-ljxkv"] Jul 24 00:50:32.002260 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:32.002178 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" path="/var/lib/kubelet/pods/77aff1e0-41e3-4853-851f-2c7b2566035c/volumes" Jul 24 00:50:39.885764 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:39.885730 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:50:47.860671 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:47.860636 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj"] Jul 24 00:50:47.861115 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:47.860908 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" podUID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerName="istio-proxy" containerID="cri-o://6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d" gracePeriod=30 Jul 24 00:50:48.804780 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.804743 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt"] Jul 24 00:50:48.805395 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.805373 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerName="istio-proxy" Jul 24 00:50:48.805448 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.805398 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerName="istio-proxy" Jul 24 00:50:48.805448 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.805422 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerName="istio-proxy" Jul 24 00:50:48.805448 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.805431 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerName="istio-proxy" Jul 24 00:50:48.805538 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.805454 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b518c9-985f-451b-8a59-eb65a018d8b3" containerName="main" Jul 24 00:50:48.805538 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.805464 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b518c9-985f-451b-8a59-eb65a018d8b3" containerName="main" Jul 24 00:50:48.805605 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.805570 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerName="istio-proxy" Jul 24 00:50:48.805605 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.805585 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="77aff1e0-41e3-4853-851f-2c7b2566035c" containerName="istio-proxy" Jul 24 00:50:48.805605 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.805594 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b518c9-985f-451b-8a59-eb65a018d8b3" containerName="main" Jul 24 00:50:48.810280 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.810259 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:48.812755 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.812736 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6a8188-openshift-default-dockercfg-ptqfp\"" Jul 24 00:50:48.823629 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.823585 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt"] Jul 24 00:50:48.932619 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.932579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-podinfo\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:48.932619 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.932622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-workload-socket\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:48.933100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.932660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istiod-ca-cert\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:48.933100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.932716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-credential-socket\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:48.933100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.932744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-token\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:48.933100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.932782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-data\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:48.933100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.932801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-workload-certs\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:48.933100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.932819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z25ff\" (UniqueName: \"kubernetes.io/projected/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-kube-api-access-z25ff\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:48.933100 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:48.932852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-envoy\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.033389 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.033350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-credential-socket\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.033389 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.033387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-token\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.033641 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.033413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-data\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.033641 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.033549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-workload-certs\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.033641 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.033589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z25ff\" (UniqueName: \"kubernetes.io/projected/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-kube-api-access-z25ff\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.033807 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.033644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-envoy\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.033807 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.033728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-podinfo\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.033807 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.033757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-workload-socket\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.033807 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.033801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istiod-ca-cert\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.034048 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.033808 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-credential-socket\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.034048 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.033816 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-data\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.034048 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.033872 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-workload-certs\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.034197 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.034175 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-workload-socket\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.034463 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.034445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istiod-ca-cert\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.036029 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.036007 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-envoy\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.036175 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.036158 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-podinfo\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.041226 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.041199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-token\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.041491 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.041462 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z25ff\" (UniqueName: \"kubernetes.io/projected/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-kube-api-access-z25ff\") pod \"e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.122792 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.122700 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:49.266819 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.266792 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt"] Jul 24 00:50:49.268817 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:50:49.268788 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06a6a42_62a0_4e6b_8511_d477e07ffb5b.slice/crio-a48c2b3b32b9ed700e95a75f818b6c84836df79489c5ef802908e0ce9c625e39 WatchSource:0}: Error finding container a48c2b3b32b9ed700e95a75f818b6c84836df79489c5ef802908e0ce9c625e39: Status 404 returned error can't find the container with id a48c2b3b32b9ed700e95a75f818b6c84836df79489c5ef802908e0ce9c625e39 Jul 24 00:50:49.271359 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.271325 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:50:49.271446 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.271394 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:50:49.271446 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.271428 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:50:49.830557 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.830518 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" event={"ID":"e06a6a42-62a0-4e6b-8511-d477e07ffb5b","Type":"ContainerStarted","Data":"8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869"} Jul 24 00:50:49.830749 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.830582 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" event={"ID":"e06a6a42-62a0-4e6b-8511-d477e07ffb5b","Type":"ContainerStarted","Data":"a48c2b3b32b9ed700e95a75f818b6c84836df79489c5ef802908e0ce9c625e39"} Jul 24 00:50:49.849559 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:49.849502 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" podStartSLOduration=1.8494841100000001 podStartE2EDuration="1.84948411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:50:49.848106847 +0000 UTC m=+2226.421203649" watchObservedRunningTime="2026-07-24 00:50:49.84948411 +0000 UTC m=+2226.422580911" Jul 24 00:50:50.123472 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:50.123377 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:50.128339 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:50.128317 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:50.836133 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:50.836103 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:50.837013 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:50.836997 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:50:53.114451 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.114427 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:50:53.170970 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.170880 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-podinfo\") pod \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " Jul 24 00:50:53.170970 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.170965 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkvkp\" (UniqueName: \"kubernetes.io/projected/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-kube-api-access-rkvkp\") pod \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " Jul 24 00:50:53.171179 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171024 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istiod-ca-cert\") pod \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " Jul 24 00:50:53.171179 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171072 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-workload-socket\") pod \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " Jul 24 00:50:53.171179 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171099 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-workload-certs\") pod \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " Jul 24 00:50:53.171179 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171134 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-token\") pod \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " Jul 24 00:50:53.171179 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171174 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-credential-socket\") pod \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " Jul 24 00:50:53.171471 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171204 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-data\") pod \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " Jul 24 00:50:53.171471 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171263 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-envoy\") pod \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\" (UID: \"181a1da7-3181-4aa7-8b44-98e0ffa9eac4\") " Jul 24 00:50:53.171471 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171395 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "181a1da7-3181-4aa7-8b44-98e0ffa9eac4" (UID: "181a1da7-3181-4aa7-8b44-98e0ffa9eac4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:53.171471 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171431 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "181a1da7-3181-4aa7-8b44-98e0ffa9eac4" (UID: "181a1da7-3181-4aa7-8b44-98e0ffa9eac4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:53.171471 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171463 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "181a1da7-3181-4aa7-8b44-98e0ffa9eac4" (UID: "181a1da7-3181-4aa7-8b44-98e0ffa9eac4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:50:53.171767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171748 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:53.171832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171765 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "181a1da7-3181-4aa7-8b44-98e0ffa9eac4" (UID: "181a1da7-3181-4aa7-8b44-98e0ffa9eac4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:53.171832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171775 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:53.171832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.171825 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:53.172204 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.172177 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-data" (OuterVolumeSpecName: "istio-data") pod "181a1da7-3181-4aa7-8b44-98e0ffa9eac4" (UID: "181a1da7-3181-4aa7-8b44-98e0ffa9eac4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:53.173276 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.173242 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "181a1da7-3181-4aa7-8b44-98e0ffa9eac4" (UID: "181a1da7-3181-4aa7-8b44-98e0ffa9eac4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 00:50:53.173374 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.173296 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-kube-api-access-rkvkp" (OuterVolumeSpecName: "kube-api-access-rkvkp") pod "181a1da7-3181-4aa7-8b44-98e0ffa9eac4" (UID: "181a1da7-3181-4aa7-8b44-98e0ffa9eac4"). InnerVolumeSpecName "kube-api-access-rkvkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:50:53.173416 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.173378 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-token" (OuterVolumeSpecName: "istio-token") pod "181a1da7-3181-4aa7-8b44-98e0ffa9eac4" (UID: "181a1da7-3181-4aa7-8b44-98e0ffa9eac4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:50:53.173452 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.173428 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "181a1da7-3181-4aa7-8b44-98e0ffa9eac4" (UID: "181a1da7-3181-4aa7-8b44-98e0ffa9eac4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:50:53.273305 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.273266 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkvkp\" (UniqueName: \"kubernetes.io/projected/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-kube-api-access-rkvkp\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:53.273305 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.273301 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:53.273305 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.273312 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:53.273767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.273324 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:53.273767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.273332 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:53.273767 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.273339 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181a1da7-3181-4aa7-8b44-98e0ffa9eac4-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:50:53.853474 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.853441 2576 generic.go:356] "Generic (PLEG): container finished" podID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerID="6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d" exitCode=0 Jul 24 00:50:53.853658 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.853491 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" Jul 24 00:50:53.853658 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.853515 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" event={"ID":"181a1da7-3181-4aa7-8b44-98e0ffa9eac4","Type":"ContainerDied","Data":"6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d"} Jul 24 00:50:53.853658 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.853563 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj" event={"ID":"181a1da7-3181-4aa7-8b44-98e0ffa9eac4","Type":"ContainerDied","Data":"3fd12c4d6be15501cdbb4ec1cf63eb7cb960cca96034e0826f024e292729ba99"} Jul 24 00:50:53.853658 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.853584 2576 scope.go:117] "RemoveContainer" containerID="6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d" Jul 24 00:50:53.863071 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.863055 2576 scope.go:117] "RemoveContainer" containerID="d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2" Jul 24 00:50:53.871694 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.871675 2576 scope.go:117] "RemoveContainer" containerID="6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d" Jul 24 00:50:53.872104 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:50:53.872082 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d\": container with ID starting with 6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d not found: ID does not exist" containerID="6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d" Jul 24 00:50:53.872161 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.872115 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d"} err="failed to get container status \"6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d\": rpc error: code = NotFound desc = could not find container \"6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d\": container with ID starting with 6e48dd5b3f16e594276eae137a7b2e344341144df638bcf79918473d08d4fa2d not found: ID does not exist" Jul 24 00:50:53.872161 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.872134 2576 scope.go:117] "RemoveContainer" containerID="d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2" Jul 24 00:50:53.872404 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:50:53.872389 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2\": container with ID starting with d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2 not found: ID does not exist" containerID="d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2" Jul 24 00:50:53.872473 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.872406 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2"} err="failed to get container status \"d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2\": rpc error: code = NotFound desc = could not find container \"d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2\": container with ID starting with d2bb69bb98831619770229ecef7712cfcfd6e577f3c34d107ee22e50521402a2 not found: ID does not exist" Jul 24 00:50:53.876186 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.876163 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj"] Jul 24 00:50:53.879829 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:53.879805 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e7050a-openshift-default-678874c656-nnqzj"] Jul 24 00:50:54.002657 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:54.002621 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" path="/var/lib/kubelet/pods/181a1da7-3181-4aa7-8b44-98e0ffa9eac4/volumes" Jul 24 00:50:56.989528 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:56.989488 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:50:57.110867 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.110830 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm"] Jul 24 00:50:57.111247 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.111232 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerName="istio-proxy" Jul 24 00:50:57.111327 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.111250 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerName="istio-proxy" Jul 24 00:50:57.111366 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.111351 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerName="istio-proxy" Jul 24 00:50:57.111366 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.111364 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerName="istio-proxy" Jul 24 00:50:57.116127 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.116096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.118472 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.118442 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a43e61-openshift-default-dockercfg-k9fwh\"" Jul 24 00:50:57.127243 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.127203 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm"] Jul 24 00:50:57.216066 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.216031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-workload-socket\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.216284 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.216071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-istio-data\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.216284 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.216131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9wrd\" (UniqueName: \"kubernetes.io/projected/90412fdb-0355-494f-b023-3663dd2faa2a-kube-api-access-l9wrd\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.216284 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.216162 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90412fdb-0355-494f-b023-3663dd2faa2a-istiod-ca-cert\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.216284 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.216213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-workload-certs\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.216284 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.216245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-credential-socket\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.216284 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.216276 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90412fdb-0355-494f-b023-3663dd2faa2a-istio-podinfo\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.216542 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.216303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90412fdb-0355-494f-b023-3663dd2faa2a-istio-token\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.216542 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.216367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-istio-envoy\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.317302 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-workload-socket\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.317302 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-istio-data\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.317567 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9wrd\" (UniqueName: \"kubernetes.io/projected/90412fdb-0355-494f-b023-3663dd2faa2a-kube-api-access-l9wrd\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.317567 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90412fdb-0355-494f-b023-3663dd2faa2a-istiod-ca-cert\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.317567 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-workload-certs\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.317567 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-credential-socket\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.317567 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90412fdb-0355-494f-b023-3663dd2faa2a-istio-podinfo\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.317865 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317602 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90412fdb-0355-494f-b023-3663dd2faa2a-istio-token\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.317865 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-istio-envoy\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.317865 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-istio-data\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.317865 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-workload-socket\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.318083 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-workload-certs\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.318083 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.317987 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-credential-socket\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.318214 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.318195 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90412fdb-0355-494f-b023-3663dd2faa2a-istiod-ca-cert\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.320017 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.319991 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-istio-envoy\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.320129 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.320113 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90412fdb-0355-494f-b023-3663dd2faa2a-istio-podinfo\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.327328 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.327297 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90412fdb-0355-494f-b023-3663dd2faa2a-istio-token\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.327533 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.327512 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9wrd\" (UniqueName: \"kubernetes.io/projected/90412fdb-0355-494f-b023-3663dd2faa2a-kube-api-access-l9wrd\") pod \"e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.429505 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.429466 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:57.577423 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.577390 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm"] Jul 24 00:50:57.578443 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:50:57.578409 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90412fdb_0355_494f_b023_3663dd2faa2a.slice/crio-faef4249992a2e48af4850cde84072620135590a57eabcf9194a197b014bd04f WatchSource:0}: Error finding container faef4249992a2e48af4850cde84072620135590a57eabcf9194a197b014bd04f: Status 404 returned error can't find the container with id faef4249992a2e48af4850cde84072620135590a57eabcf9194a197b014bd04f Jul 24 00:50:57.580690 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.580652 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:50:57.580782 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.580763 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:50:57.580832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.580805 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:50:57.879381 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.879282 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" event={"ID":"90412fdb-0355-494f-b023-3663dd2faa2a","Type":"ContainerStarted","Data":"eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c"} Jul 24 00:50:57.879381 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.879360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" event={"ID":"90412fdb-0355-494f-b023-3663dd2faa2a","Type":"ContainerStarted","Data":"faef4249992a2e48af4850cde84072620135590a57eabcf9194a197b014bd04f"} Jul 24 00:50:57.900365 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:57.900304 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" podStartSLOduration=0.900282358 podStartE2EDuration="900.282358ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:50:57.898241937 +0000 UTC m=+2234.471338739" watchObservedRunningTime="2026-07-24 00:50:57.900282358 +0000 UTC m=+2234.473379162" Jul 24 00:50:58.430047 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:58.429997 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:58.442507 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:58.442473 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:58.885533 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:58.885504 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:50:58.886631 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:50:58.886611 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:51:04.925196 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:04.925106 2576 generic.go:356] "Generic (PLEG): container finished" podID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerID="8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869" exitCode=0 Jul 24 00:51:04.925585 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:04.925185 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" event={"ID":"e06a6a42-62a0-4e6b-8511-d477e07ffb5b","Type":"ContainerDied","Data":"8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869"} Jul 24 00:51:04.925585 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:04.925580 2576 scope.go:117] "RemoveContainer" containerID="8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869" Jul 24 00:51:04.926613 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:04.926586 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:51:04.926668 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:04.926654 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:51:04.926715 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:04.926704 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:51:05.836931 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:05.836884 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:51:05.933026 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:05.932987 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" event={"ID":"e06a6a42-62a0-4e6b-8511-d477e07ffb5b","Type":"ContainerStarted","Data":"5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926"} Jul 24 00:51:06.123456 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:06.123364 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:51:06.124802 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:06.124768 2576 patch_prober.go:28] interesting pod/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 24 00:51:06.124944 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:06.124817 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 24 00:51:07.124006 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:07.123969 2576 patch_prober.go:28] interesting pod/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 24 00:51:07.124406 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:07.124030 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 24 00:51:07.196131 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:07.196086 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:51:08.123906 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:08.123863 2576 patch_prober.go:28] interesting pod/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 24 00:51:08.124168 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:08.123933 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 24 00:51:09.123203 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:09.123164 2576 patch_prober.go:28] interesting pod/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 24 00:51:09.123406 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:09.123255 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 24 00:51:10.127138 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:10.127106 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:51:10.127600 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:10.127480 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:51:10.128253 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:10.128234 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:51:14.928102 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:14.928065 2576 patch_prober.go:28] interesting pod/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 00:51:14.928512 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:14.928131 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 00:51:16.004913 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:16.004880 2576 generic.go:356] "Generic (PLEG): container finished" podID="90412fdb-0355-494f-b023-3663dd2faa2a" containerID="eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c" exitCode=0 Jul 24 00:51:16.005349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:16.004964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" event={"ID":"90412fdb-0355-494f-b023-3663dd2faa2a","Type":"ContainerDied","Data":"eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c"} Jul 24 00:51:16.005349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:16.005343 2576 scope.go:117] "RemoveContainer" containerID="eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c" Jul 24 00:51:16.006380 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:16.006349 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:51:16.006439 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:16.006420 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:51:16.006474 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:16.006455 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:51:17.016106 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:17.016067 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" event={"ID":"90412fdb-0355-494f-b023-3663dd2faa2a","Type":"ContainerStarted","Data":"6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88"} Jul 24 00:51:17.429879 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:17.429829 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:51:17.430179 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:17.430156 2576 patch_prober.go:28] interesting pod/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 24 00:51:17.430248 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:17.430208 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 24 00:51:18.430036 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:18.430001 2576 patch_prober.go:28] interesting pod/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 24 00:51:18.430415 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:18.430058 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 24 00:51:19.429913 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:19.429874 2576 patch_prober.go:28] interesting pod/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 24 00:51:19.430118 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:19.430001 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 24 00:51:20.430537 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:20.430492 2576 patch_prober.go:28] interesting pod/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 24 00:51:20.430980 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:20.430555 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 24 00:51:21.433970 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:21.433918 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:51:21.434366 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:21.434345 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:51:21.435098 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:21.435077 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:51:31.085433 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:31.085393 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:51:39.390943 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:39.390900 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt"] Jul 24 00:51:39.391534 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:39.391220 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerName="istio-proxy" containerID="cri-o://5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926" gracePeriod=30 Jul 24 00:51:40.130319 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:40.130282 2576 patch_prober.go:28] interesting pod/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 24 00:51:40.130546 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:40.130344 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 00:51:44.258166 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.258133 2576 scope.go:117] "RemoveContainer" containerID="ef3268de8b12bd630c1afa7e8fd0c5cfc676b7aaf73ae16b6398dee9589fe74c" Jul 24 00:51:44.267015 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.266993 2576 scope.go:117] "RemoveContainer" containerID="284b64e21e71cedd0b7ad9f98d713f2bbbef96a5829960b5143e171a9966179c" Jul 24 00:51:44.660665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.660640 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:51:44.771122 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.771022 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z25ff\" (UniqueName: \"kubernetes.io/projected/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-kube-api-access-z25ff\") pod \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " Jul 24 00:51:44.771122 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.771105 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-data\") pod \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " Jul 24 00:51:44.771381 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.771131 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-podinfo\") pod \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " Jul 24 00:51:44.771381 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.771148 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-token\") pod \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " Jul 24 00:51:44.771381 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.771298 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istiod-ca-cert\") pod \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " Jul 24 00:51:44.771381 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.771345 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-envoy\") pod \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " Jul 24 00:51:44.771603 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.771411 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-workload-socket\") pod \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " Jul 24 00:51:44.771603 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.771437 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-credential-socket\") pod \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " Jul 24 00:51:44.771603 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.771463 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-workload-certs\") pod \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\" (UID: \"e06a6a42-62a0-4e6b-8511-d477e07ffb5b\") " Jul 24 00:51:44.771759 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.771675 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e06a6a42-62a0-4e6b-8511-d477e07ffb5b" (UID: "e06a6a42-62a0-4e6b-8511-d477e07ffb5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:51:44.771901 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.771879 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:51:44.771985 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.771880 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e06a6a42-62a0-4e6b-8511-d477e07ffb5b" (UID: "e06a6a42-62a0-4e6b-8511-d477e07ffb5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:51:44.772094 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.772074 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e06a6a42-62a0-4e6b-8511-d477e07ffb5b" (UID: "e06a6a42-62a0-4e6b-8511-d477e07ffb5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:51:44.772156 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.772117 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e06a6a42-62a0-4e6b-8511-d477e07ffb5b" (UID: "e06a6a42-62a0-4e6b-8511-d477e07ffb5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:51:44.772213 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.772183 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-data" (OuterVolumeSpecName: "istio-data") pod "e06a6a42-62a0-4e6b-8511-d477e07ffb5b" (UID: "e06a6a42-62a0-4e6b-8511-d477e07ffb5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:51:44.773551 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.773520 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e06a6a42-62a0-4e6b-8511-d477e07ffb5b" (UID: "e06a6a42-62a0-4e6b-8511-d477e07ffb5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 00:51:44.773551 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.773530 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-kube-api-access-z25ff" (OuterVolumeSpecName: "kube-api-access-z25ff") pod "e06a6a42-62a0-4e6b-8511-d477e07ffb5b" (UID: "e06a6a42-62a0-4e6b-8511-d477e07ffb5b"). InnerVolumeSpecName "kube-api-access-z25ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:51:44.773747 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.773590 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-token" (OuterVolumeSpecName: "istio-token") pod "e06a6a42-62a0-4e6b-8511-d477e07ffb5b" (UID: "e06a6a42-62a0-4e6b-8511-d477e07ffb5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:51:44.773747 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.773712 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e06a6a42-62a0-4e6b-8511-d477e07ffb5b" (UID: "e06a6a42-62a0-4e6b-8511-d477e07ffb5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:51:44.872846 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.872803 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:51:44.872846 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.872839 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:51:44.872846 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.872849 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z25ff\" (UniqueName: \"kubernetes.io/projected/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-kube-api-access-z25ff\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:51:44.872846 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.872858 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:51:44.873342 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.872867 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:51:44.873342 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.872874 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:51:44.873342 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.872897 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:51:44.873342 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:44.872905 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06a6a42-62a0-4e6b-8511-d477e07ffb5b-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:51:45.185315 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:45.185283 2576 generic.go:356] "Generic (PLEG): container finished" podID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerID="5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926" exitCode=0 Jul 24 00:51:45.185525 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:45.185330 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" Jul 24 00:51:45.185525 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:45.185369 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" event={"ID":"e06a6a42-62a0-4e6b-8511-d477e07ffb5b","Type":"ContainerDied","Data":"5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926"} Jul 24 00:51:45.185525 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:45.185431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt" event={"ID":"e06a6a42-62a0-4e6b-8511-d477e07ffb5b","Type":"ContainerDied","Data":"a48c2b3b32b9ed700e95a75f818b6c84836df79489c5ef802908e0ce9c625e39"} Jul 24 00:51:45.185525 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:45.185467 2576 scope.go:117] "RemoveContainer" containerID="5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926" Jul 24 00:51:45.195728 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:45.195710 2576 scope.go:117] "RemoveContainer" containerID="8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869" Jul 24 00:51:45.205171 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:45.205147 2576 scope.go:117] "RemoveContainer" containerID="5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926" Jul 24 00:51:45.205494 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:51:45.205474 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926\": container with ID starting with 5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926 not found: ID does not exist" containerID="5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926" Jul 24 00:51:45.205550 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:45.205504 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926"} err="failed to get container status \"5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926\": rpc error: code = NotFound desc = could not find container \"5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926\": container with ID starting with 5cb76cda91d3c4ce85525810f97f658c2da3470dfe24871b5c7c1a9051ad0926 not found: ID does not exist" Jul 24 00:51:45.205550 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:45.205525 2576 scope.go:117] "RemoveContainer" containerID="8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869" Jul 24 00:51:45.205767 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:51:45.205743 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869\": container with ID starting with 8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869 not found: ID does not exist" containerID="8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869" Jul 24 00:51:45.205809 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:45.205775 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869"} err="failed to get container status \"8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869\": rpc error: code = NotFound desc = could not find container \"8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869\": container with ID starting with 8122935a57c6340ecc402eee1e1774fc4a448c55705ac73768531af957c26869 not found: ID does not exist" Jul 24 00:51:45.210257 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:45.210227 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt"] Jul 24 00:51:45.216301 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:45.216274 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6a8188-openshift-default-5dc6f99ffb-vxlwt"] Jul 24 00:51:46.001717 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:46.001666 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" path="/var/lib/kubelet/pods/e06a6a42-62a0-4e6b-8511-d477e07ffb5b/volumes" Jul 24 00:51:59.789156 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:51:59.789118 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:52:06.366821 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:06.366780 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm"] Jul 24 00:52:06.367277 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:06.367077 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" containerID="cri-o://6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88" gracePeriod=30 Jul 24 00:52:06.437209 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:06.437170 2576 patch_prober.go:28] interesting pod/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 00:52:06.437396 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:06.437236 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 00:52:07.307377 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.307341 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9"] Jul 24 00:52:07.307867 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.307847 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerName="istio-proxy" Jul 24 00:52:07.307867 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.307870 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="181a1da7-3181-4aa7-8b44-98e0ffa9eac4" containerName="istio-proxy" Jul 24 00:52:07.308091 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.307891 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerName="istio-proxy" Jul 24 00:52:07.308091 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.307899 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerName="istio-proxy" Jul 24 00:52:07.308091 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.308015 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerName="istio-proxy" Jul 24 00:52:07.308091 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.308026 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerName="istio-proxy" Jul 24 00:52:07.311318 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.311294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.313906 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.313885 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-38da25-openshift-default-dockercfg-qcwnw\"" Jul 24 00:52:07.323184 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.323158 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9"] Jul 24 00:52:07.474855 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.474817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-workload-socket\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.475332 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.474864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-workload-certs\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.475332 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.474921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-data\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.475332 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.474968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz8jz\" (UniqueName: \"kubernetes.io/projected/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-kube-api-access-kz8jz\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.475332 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.475004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-token\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.475332 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.475072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-envoy\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.475332 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.475123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istiod-ca-cert\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.475332 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.475159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-credential-socket\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.475332 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.475175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-podinfo\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.575981 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.575865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-workload-socket\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.575981 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.575907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-workload-certs\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.575981 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.575930 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-data\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.576269 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.575986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz8jz\" (UniqueName: \"kubernetes.io/projected/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-kube-api-access-kz8jz\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.576269 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.576020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-token\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.576269 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.576052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-envoy\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.576269 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.576097 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istiod-ca-cert\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.576269 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.576141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-credential-socket\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.576269 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.576164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-podinfo\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.576591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.576374 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-workload-certs\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.576591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.576456 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-data\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.576591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.576557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-credential-socket\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.576701 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.576644 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-workload-socket\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.576876 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.576857 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istiod-ca-cert\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.578321 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.578299 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-envoy\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.578731 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.578715 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-podinfo\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.584319 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.584292 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-token\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.585057 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.585040 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz8jz\" (UniqueName: \"kubernetes.io/projected/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-kube-api-access-kz8jz\") pod \"e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.624125 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.624094 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:07.767088 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.767054 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9"] Jul 24 00:52:07.768621 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:52:07.768593 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5e619d_5ec8_4078_bee2_4cdf7addf4b1.slice/crio-642a9491f4cb497977395fa5ce8534e732caf27868aa2ee0b761770858f8d368 WatchSource:0}: Error finding container 642a9491f4cb497977395fa5ce8534e732caf27868aa2ee0b761770858f8d368: Status 404 returned error can't find the container with id 642a9491f4cb497977395fa5ce8534e732caf27868aa2ee0b761770858f8d368 Jul 24 00:52:07.770980 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.770928 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:52:07.771050 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.771012 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:52:07.771090 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:07.771048 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:52:08.316269 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:08.316223 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" event={"ID":"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1","Type":"ContainerStarted","Data":"3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138"} Jul 24 00:52:08.316497 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:08.316292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" event={"ID":"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1","Type":"ContainerStarted","Data":"642a9491f4cb497977395fa5ce8534e732caf27868aa2ee0b761770858f8d368"} Jul 24 00:52:08.336329 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:08.336273 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" podStartSLOduration=1.336257057 podStartE2EDuration="1.336257057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:52:08.335176364 +0000 UTC m=+2304.908273166" watchObservedRunningTime="2026-07-24 00:52:08.336257057 +0000 UTC m=+2304.909353854" Jul 24 00:52:08.624804 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:08.624707 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:08.630068 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:08.630030 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:09.321772 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:09.321735 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:09.322797 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:09.322773 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:11.629093 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.629028 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:52:11.717656 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.717622 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-workload-socket\") pod \"90412fdb-0355-494f-b023-3663dd2faa2a\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " Jul 24 00:52:11.717656 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.717660 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-credential-socket\") pod \"90412fdb-0355-494f-b023-3663dd2faa2a\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " Jul 24 00:52:11.717920 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.717680 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-istio-data\") pod \"90412fdb-0355-494f-b023-3663dd2faa2a\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " Jul 24 00:52:11.717920 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.717723 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90412fdb-0355-494f-b023-3663dd2faa2a-istio-token\") pod \"90412fdb-0355-494f-b023-3663dd2faa2a\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " Jul 24 00:52:11.717920 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.717744 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-workload-certs\") pod \"90412fdb-0355-494f-b023-3663dd2faa2a\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " Jul 24 00:52:11.717920 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.717788 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90412fdb-0355-494f-b023-3663dd2faa2a-istiod-ca-cert\") pod \"90412fdb-0355-494f-b023-3663dd2faa2a\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " Jul 24 00:52:11.717920 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.717836 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9wrd\" (UniqueName: \"kubernetes.io/projected/90412fdb-0355-494f-b023-3663dd2faa2a-kube-api-access-l9wrd\") pod \"90412fdb-0355-494f-b023-3663dd2faa2a\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " Jul 24 00:52:11.717920 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.717889 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-istio-envoy\") pod \"90412fdb-0355-494f-b023-3663dd2faa2a\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " Jul 24 00:52:11.718241 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.717934 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90412fdb-0355-494f-b023-3663dd2faa2a-istio-podinfo\") pod \"90412fdb-0355-494f-b023-3663dd2faa2a\" (UID: \"90412fdb-0355-494f-b023-3663dd2faa2a\") " Jul 24 00:52:11.718303 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.717940 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "90412fdb-0355-494f-b023-3663dd2faa2a" (UID: "90412fdb-0355-494f-b023-3663dd2faa2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:52:11.718303 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.718259 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "90412fdb-0355-494f-b023-3663dd2faa2a" (UID: "90412fdb-0355-494f-b023-3663dd2faa2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:52:11.718436 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.718409 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "90412fdb-0355-494f-b023-3663dd2faa2a" (UID: "90412fdb-0355-494f-b023-3663dd2faa2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:52:11.718659 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.718591 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90412fdb-0355-494f-b023-3663dd2faa2a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "90412fdb-0355-494f-b023-3663dd2faa2a" (UID: "90412fdb-0355-494f-b023-3663dd2faa2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:52:11.719201 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.719172 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-istio-data" (OuterVolumeSpecName: "istio-data") pod "90412fdb-0355-494f-b023-3663dd2faa2a" (UID: "90412fdb-0355-494f-b023-3663dd2faa2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:52:11.720350 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.720329 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90412fdb-0355-494f-b023-3663dd2faa2a-istio-token" (OuterVolumeSpecName: "istio-token") pod "90412fdb-0355-494f-b023-3663dd2faa2a" (UID: "90412fdb-0355-494f-b023-3663dd2faa2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:52:11.720350 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.720341 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90412fdb-0355-494f-b023-3663dd2faa2a-kube-api-access-l9wrd" (OuterVolumeSpecName: "kube-api-access-l9wrd") pod "90412fdb-0355-494f-b023-3663dd2faa2a" (UID: "90412fdb-0355-494f-b023-3663dd2faa2a"). InnerVolumeSpecName "kube-api-access-l9wrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:52:11.720570 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.720554 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90412fdb-0355-494f-b023-3663dd2faa2a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "90412fdb-0355-494f-b023-3663dd2faa2a" (UID: "90412fdb-0355-494f-b023-3663dd2faa2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 00:52:11.720570 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.720562 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "90412fdb-0355-494f-b023-3663dd2faa2a" (UID: "90412fdb-0355-494f-b023-3663dd2faa2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:52:11.819339 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.819303 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:52:11.819339 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.819334 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90412fdb-0355-494f-b023-3663dd2faa2a-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:52:11.819339 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.819345 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:52:11.819596 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.819354 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90412fdb-0355-494f-b023-3663dd2faa2a-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:52:11.819596 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.819363 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9wrd\" (UniqueName: \"kubernetes.io/projected/90412fdb-0355-494f-b023-3663dd2faa2a-kube-api-access-l9wrd\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:52:11.819596 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.819373 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:52:11.819596 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.819381 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90412fdb-0355-494f-b023-3663dd2faa2a-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:52:11.819596 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.819389 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:52:11.819596 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:11.819398 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90412fdb-0355-494f-b023-3663dd2faa2a-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:52:12.340498 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:12.340462 2576 generic.go:356] "Generic (PLEG): container finished" podID="90412fdb-0355-494f-b023-3663dd2faa2a" containerID="6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88" exitCode=0 Jul 24 00:52:12.340678 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:12.340518 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" Jul 24 00:52:12.340678 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:12.340553 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" event={"ID":"90412fdb-0355-494f-b023-3663dd2faa2a","Type":"ContainerDied","Data":"6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88"} Jul 24 00:52:12.340678 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:12.340612 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm" event={"ID":"90412fdb-0355-494f-b023-3663dd2faa2a","Type":"ContainerDied","Data":"faef4249992a2e48af4850cde84072620135590a57eabcf9194a197b014bd04f"} Jul 24 00:52:12.340678 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:12.340643 2576 scope.go:117] "RemoveContainer" containerID="6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88" Jul 24 00:52:12.350155 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:12.350126 2576 scope.go:117] "RemoveContainer" containerID="eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c" Jul 24 00:52:12.358460 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:12.358423 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm"] Jul 24 00:52:12.360238 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:12.360215 2576 scope.go:117] "RemoveContainer" containerID="6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88" Jul 24 00:52:12.360607 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:52:12.360587 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88\": container with ID starting with 6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88 not found: ID does not exist" containerID="6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88" Jul 24 00:52:12.360662 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:12.360617 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88"} err="failed to get container status \"6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88\": rpc error: code = NotFound desc = could not find container \"6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88\": container with ID starting with 6ac8f50162d7fb63d2603430d41ed8b7911bb8400315f660d139c3961cebdd88 not found: ID does not exist" Jul 24 00:52:12.360662 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:12.360637 2576 scope.go:117] "RemoveContainer" containerID="eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c" Jul 24 00:52:12.360978 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:52:12.360942 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c\": container with ID starting with eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c not found: ID does not exist" containerID="eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c" Jul 24 00:52:12.361020 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:12.360985 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c"} err="failed to get container status \"eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c\": rpc error: code = NotFound desc = could not find container \"eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c\": container with ID starting with eff8864491e57efd9b7ccd6ecd17e3d6c37499c3c21196a48fcd79aa62f8f41c not found: ID does not exist" Jul 24 00:52:12.362515 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:12.362495 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a43e61-openshift-default-56b7844db-c9rwm"] Jul 24 00:52:14.001990 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:14.001935 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" path="/var/lib/kubelet/pods/90412fdb-0355-494f-b023-3663dd2faa2a/volumes" Jul 24 00:52:15.454311 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.454272 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp"] Jul 24 00:52:15.454838 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.454818 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerName="istio-proxy" Jul 24 00:52:15.454886 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.454843 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e06a6a42-62a0-4e6b-8511-d477e07ffb5b" containerName="istio-proxy" Jul 24 00:52:15.454886 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.454867 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" Jul 24 00:52:15.454886 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.454876 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" Jul 24 00:52:15.455006 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.454889 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" Jul 24 00:52:15.455006 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.454899 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" Jul 24 00:52:15.455075 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.455037 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" Jul 24 00:52:15.455075 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.455052 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90412fdb-0355-494f-b023-3663dd2faa2a" containerName="istio-proxy" Jul 24 00:52:15.460158 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.460130 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.462548 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.462521 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-44edfc-openshift-default-dockercfg-9gj8n\"" Jul 24 00:52:15.469571 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.469543 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp"] Jul 24 00:52:15.554386 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.554342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdfb9b3-1339-4b79-88de-5591c8520b57-istiod-ca-cert\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.554591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.554440 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-data\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.554591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.554477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-token\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.554591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.554536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-credential-socket\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.554591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.554558 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-workload-certs\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.554591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.554590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpnmq\" (UniqueName: \"kubernetes.io/projected/dcdfb9b3-1339-4b79-88de-5591c8520b57-kube-api-access-jpnmq\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.554873 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.554618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-envoy\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.554873 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.554653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-podinfo\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.554873 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.554680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-workload-socket\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.655343 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.655298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-credential-socket\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.655343 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.655346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-workload-certs\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.655587 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.655470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpnmq\" (UniqueName: \"kubernetes.io/projected/dcdfb9b3-1339-4b79-88de-5591c8520b57-kube-api-access-jpnmq\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.655587 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.655515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-envoy\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.655587 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.655551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-podinfo\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.655587 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.655573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-workload-socket\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.655788 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.655615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdfb9b3-1339-4b79-88de-5591c8520b57-istiod-ca-cert\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.655788 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.655708 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-credential-socket\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.655788 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.655748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-workload-certs\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.655928 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.655871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-data\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.655928 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.655917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-token\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.656064 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.655977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-workload-socket\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.656190 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.656170 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-data\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.656343 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.656321 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdfb9b3-1339-4b79-88de-5591c8520b57-istiod-ca-cert\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.658414 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.658393 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-envoy\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.658662 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.658641 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-podinfo\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.664123 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.664095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpnmq\" (UniqueName: \"kubernetes.io/projected/dcdfb9b3-1339-4b79-88de-5591c8520b57-kube-api-access-jpnmq\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.664234 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.664142 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-token\") pod \"e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.772331 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.772226 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:15.920826 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.920796 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp"] Jul 24 00:52:15.923648 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:52:15.923623 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdfb9b3_1339_4b79_88de_5591c8520b57.slice/crio-89096e0938a253341b5b233f939a53cf55d28030a4cca970932909e35f705d4f WatchSource:0}: Error finding container 89096e0938a253341b5b233f939a53cf55d28030a4cca970932909e35f705d4f: Status 404 returned error can't find the container with id 89096e0938a253341b5b233f939a53cf55d28030a4cca970932909e35f705d4f Jul 24 00:52:15.925818 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.925790 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:52:15.925895 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.925857 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:52:15.925895 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:15.925884 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:52:16.367789 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:16.367751 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" event={"ID":"dcdfb9b3-1339-4b79-88de-5591c8520b57","Type":"ContainerStarted","Data":"938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f"} Jul 24 00:52:16.368011 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:16.367814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" event={"ID":"dcdfb9b3-1339-4b79-88de-5591c8520b57","Type":"ContainerStarted","Data":"89096e0938a253341b5b233f939a53cf55d28030a4cca970932909e35f705d4f"} Jul 24 00:52:16.390190 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:16.390137 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" podStartSLOduration=1.39011889 podStartE2EDuration="1.39011889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:52:16.386855222 +0000 UTC m=+2312.959952025" watchObservedRunningTime="2026-07-24 00:52:16.39011889 +0000 UTC m=+2312.963215693" Jul 24 00:52:16.772850 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:16.772766 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:16.778160 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:16.778133 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:17.373990 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:17.373938 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:17.374909 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:17.374890 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:18.090193 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:18.090150 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:52:25.363152 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:25.363106 2576 patch_prober.go:28] interesting pod/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 00:52:25.363562 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:25.363182 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 00:52:26.428137 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:26.428104 2576 generic.go:356] "Generic (PLEG): container finished" podID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerID="3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138" exitCode=0 Jul 24 00:52:26.428597 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:26.428186 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" event={"ID":"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1","Type":"ContainerDied","Data":"3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138"} Jul 24 00:52:26.428645 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:26.428608 2576 scope.go:117] "RemoveContainer" containerID="3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138" Jul 24 00:52:26.429682 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:26.429654 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:52:26.429757 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:26.429727 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:52:26.429812 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:26.429760 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:52:27.435860 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:27.435812 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" event={"ID":"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1","Type":"ContainerStarted","Data":"2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0"} Jul 24 00:52:27.624904 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:27.624866 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:27.624904 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:27.624899 2576 patch_prober.go:28] interesting pod/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 24 00:52:27.625203 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:27.624941 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 24 00:52:28.625268 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:28.625232 2576 patch_prober.go:28] interesting pod/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 24 00:52:28.625702 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:28.625287 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 24 00:52:29.109069 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:29.109029 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:52:29.625282 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:29.625245 2576 patch_prober.go:28] interesting pod/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 24 00:52:29.625680 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:29.625297 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 24 00:52:30.625168 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:30.625129 2576 patch_prober.go:28] interesting pod/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 24 00:52:30.625358 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:30.625186 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 24 00:52:31.628608 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:31.628565 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:31.629143 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:31.629124 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:31.629971 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:31.629936 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 00:52:32.999744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:32.998672 2576 patch_prober.go:28] interesting pod/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": read tcp 10.133.0.2:47804->10.133.0.66:15021: read: connection reset by peer" start-of-body= Jul 24 00:52:32.999744 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:32.998749 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": read tcp 10.133.0.2:47804->10.133.0.66:15021: read: connection reset by peer" Jul 24 00:52:33.481009 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:33.480974 2576 generic.go:356] "Generic (PLEG): container finished" podID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerID="938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f" exitCode=0 Jul 24 00:52:33.481194 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:33.481021 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" event={"ID":"dcdfb9b3-1339-4b79-88de-5591c8520b57","Type":"ContainerDied","Data":"938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f"} Jul 24 00:52:33.481500 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:33.481483 2576 scope.go:117] "RemoveContainer" containerID="938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f" Jul 24 00:52:33.482558 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:33.482527 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:52:33.482629 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:33.482611 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:52:33.482665 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:33.482657 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:52:34.489344 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:34.489309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" event={"ID":"dcdfb9b3-1339-4b79-88de-5591c8520b57","Type":"ContainerStarted","Data":"bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84"} Jul 24 00:52:34.773391 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:34.773297 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:34.773592 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:34.773566 2576 patch_prober.go:28] interesting pod/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 24 00:52:34.773654 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:34.773634 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 24 00:52:35.773380 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:35.773335 2576 patch_prober.go:28] interesting pod/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 24 00:52:35.773804 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:35.773408 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 24 00:52:36.773230 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:36.773147 2576 patch_prober.go:28] interesting pod/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 24 00:52:36.773230 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:36.773217 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 24 00:52:37.772931 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:37.772885 2576 patch_prober.go:28] interesting pod/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 24 00:52:37.773367 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:37.772983 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 24 00:52:38.777049 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:38.777018 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:38.777434 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:38.777356 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:38.778134 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:38.778114 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 00:52:42.390172 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:42.390136 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:52:44.295806 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:52:44.295772 2576 scope.go:117] "RemoveContainer" containerID="56e301962559903aaf0c390783a278a1138059876ea3888222d6967592b7ef51" Jul 24 00:53:10.006430 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.006391 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk"] Jul 24 00:53:10.010168 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.010143 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.013655 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.013621 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-44edfc\"/\"openshift-service-ca.crt\"" Jul 24 00:53:10.013655 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.013624 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-44edfc\"/\"kube-root-ca.crt\"" Jul 24 00:53:10.014625 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.014606 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-44edfc\"/\"default-dockercfg-848fn\"" Jul 24 00:53:10.014719 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.014700 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-44edfc\"/\"auth-test-model-44edfc-kserve-self-signed-certs\"" Jul 24 00:53:10.019971 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.019930 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk"] Jul 24 00:53:10.086236 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.086201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7975e2-bd3e-4007-87ca-f0875322ce62-tls-certs\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.086236 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.086243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-model-cache\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.086517 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.086268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-home\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.086517 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.086413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htmg4\" (UniqueName: \"kubernetes.io/projected/8c7975e2-bd3e-4007-87ca-f0875322ce62-kube-api-access-htmg4\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.086517 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.086438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-dshm\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.086517 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.086466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-tmp-dir\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.187840 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.187795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmg4\" (UniqueName: \"kubernetes.io/projected/8c7975e2-bd3e-4007-87ca-f0875322ce62-kube-api-access-htmg4\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.187840 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.187844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-dshm\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.188155 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.187866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-tmp-dir\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.188155 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.187916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7975e2-bd3e-4007-87ca-f0875322ce62-tls-certs\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.188155 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.187934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-model-cache\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.188155 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.187978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-home\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.188396 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.188369 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-tmp-dir\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.188446 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.188419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-model-cache\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.188446 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.188433 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-home\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.190262 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.190237 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-dshm\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.190432 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.190415 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7975e2-bd3e-4007-87ca-f0875322ce62-tls-certs\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.196625 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.196599 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htmg4\" (UniqueName: \"kubernetes.io/projected/8c7975e2-bd3e-4007-87ca-f0875322ce62-kube-api-access-htmg4\") pod \"auth-test-model-44edfc-kserve-b675b9f47-22jzk\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.321114 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.321058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:10.489650 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.489623 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk"] Jul 24 00:53:10.491166 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:53:10.491135 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c7975e2_bd3e_4007_87ca_f0875322ce62.slice/crio-0e8b14d3b3e835becd54701774650d77de8254de319abca5e82d1efea82da123 WatchSource:0}: Error finding container 0e8b14d3b3e835becd54701774650d77de8254de319abca5e82d1efea82da123: Status 404 returned error can't find the container with id 0e8b14d3b3e835becd54701774650d77de8254de319abca5e82d1efea82da123 Jul 24 00:53:10.700105 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.700008 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" event={"ID":"8c7975e2-bd3e-4007-87ca-f0875322ce62","Type":"ContainerStarted","Data":"c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87"} Jul 24 00:53:10.700105 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.700055 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" event={"ID":"8c7975e2-bd3e-4007-87ca-f0875322ce62","Type":"ContainerStarted","Data":"0e8b14d3b3e835becd54701774650d77de8254de319abca5e82d1efea82da123"} Jul 24 00:53:10.719583 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:10.719514 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" podStartSLOduration=1.719498791 podStartE2EDuration="1.719498791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:53:10.71656491 +0000 UTC m=+2367.289661712" watchObservedRunningTime="2026-07-24 00:53:10.719498791 +0000 UTC m=+2367.292595595" Jul 24 00:53:13.486390 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:13.486351 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:53:20.321428 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:20.321382 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:20.321858 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:20.321452 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:20.334310 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:20.334269 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:20.768711 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:20.768615 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:53:29.697436 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:29.697393 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:53:44.137796 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:44.137768 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:53:44.145170 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:44.145147 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:53:48.783564 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:53:48.783515 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:54:04.585966 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:54:04.585910 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:54:23.895718 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:54:23.895679 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:54:42.692146 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:54:42.692101 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:54:57.192837 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:54:57.192797 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:55:10.006803 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:10.006756 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk"] Jul 24 00:55:13.189112 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:13.189065 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:55:25.027246 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.027200 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" podUID="8c7975e2-bd3e-4007-87ca-f0875322ce62" containerName="main" containerID="cri-o://c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87" gracePeriod=45 Jul 24 00:55:25.290580 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.290557 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:55:25.376381 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.376349 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7975e2-bd3e-4007-87ca-f0875322ce62-tls-certs\") pod \"8c7975e2-bd3e-4007-87ca-f0875322ce62\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " Jul 24 00:55:25.376577 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.376395 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-model-cache\") pod \"8c7975e2-bd3e-4007-87ca-f0875322ce62\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " Jul 24 00:55:25.376577 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.376456 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-tmp-dir\") pod \"8c7975e2-bd3e-4007-87ca-f0875322ce62\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " Jul 24 00:55:25.376577 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.376475 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-home\") pod \"8c7975e2-bd3e-4007-87ca-f0875322ce62\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " Jul 24 00:55:25.376577 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.376511 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-dshm\") pod \"8c7975e2-bd3e-4007-87ca-f0875322ce62\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " Jul 24 00:55:25.376577 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.376556 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htmg4\" (UniqueName: \"kubernetes.io/projected/8c7975e2-bd3e-4007-87ca-f0875322ce62-kube-api-access-htmg4\") pod \"8c7975e2-bd3e-4007-87ca-f0875322ce62\" (UID: \"8c7975e2-bd3e-4007-87ca-f0875322ce62\") " Jul 24 00:55:25.376839 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.376758 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c7975e2-bd3e-4007-87ca-f0875322ce62" (UID: "8c7975e2-bd3e-4007-87ca-f0875322ce62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:55:25.376839 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.376773 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c7975e2-bd3e-4007-87ca-f0875322ce62" (UID: "8c7975e2-bd3e-4007-87ca-f0875322ce62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:55:25.376839 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.376790 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-home" (OuterVolumeSpecName: "home") pod "8c7975e2-bd3e-4007-87ca-f0875322ce62" (UID: "8c7975e2-bd3e-4007-87ca-f0875322ce62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:55:25.377035 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.376902 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-model-cache\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:55:25.377035 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.376914 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-tmp-dir\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:55:25.377035 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.376922 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-home\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:55:25.378701 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.378667 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-dshm" (OuterVolumeSpecName: "dshm") pod "8c7975e2-bd3e-4007-87ca-f0875322ce62" (UID: "8c7975e2-bd3e-4007-87ca-f0875322ce62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:55:25.378808 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.378701 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7975e2-bd3e-4007-87ca-f0875322ce62-kube-api-access-htmg4" (OuterVolumeSpecName: "kube-api-access-htmg4") pod "8c7975e2-bd3e-4007-87ca-f0875322ce62" (UID: "8c7975e2-bd3e-4007-87ca-f0875322ce62"). InnerVolumeSpecName "kube-api-access-htmg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:55:25.378808 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.378762 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7975e2-bd3e-4007-87ca-f0875322ce62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c7975e2-bd3e-4007-87ca-f0875322ce62" (UID: "8c7975e2-bd3e-4007-87ca-f0875322ce62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:55:25.453502 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.453462 2576 generic.go:356] "Generic (PLEG): container finished" podID="8c7975e2-bd3e-4007-87ca-f0875322ce62" containerID="c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87" exitCode=0 Jul 24 00:55:25.453502 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.453507 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" Jul 24 00:55:25.453711 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.453533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" event={"ID":"8c7975e2-bd3e-4007-87ca-f0875322ce62","Type":"ContainerDied","Data":"c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87"} Jul 24 00:55:25.453711 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.453570 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk" event={"ID":"8c7975e2-bd3e-4007-87ca-f0875322ce62","Type":"ContainerDied","Data":"0e8b14d3b3e835becd54701774650d77de8254de319abca5e82d1efea82da123"} Jul 24 00:55:25.453711 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.453591 2576 scope.go:117] "RemoveContainer" containerID="c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87" Jul 24 00:55:25.465205 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.465181 2576 scope.go:117] "RemoveContainer" containerID="c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87" Jul 24 00:55:25.465530 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:55:25.465508 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87\": container with ID starting with c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87 not found: ID does not exist" containerID="c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87" Jul 24 00:55:25.465588 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.465540 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87"} err="failed to get container status \"c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87\": rpc error: code = NotFound desc = could not find container \"c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87\": container with ID starting with c1eea0e442152d9e9c91edfa34e98091be66bfbfa9be5fc6e2bf7eddcc4a2e87 not found: ID does not exist" Jul 24 00:55:25.475535 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.475499 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk"] Jul 24 00:55:25.478336 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.478308 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c7975e2-bd3e-4007-87ca-f0875322ce62-dshm\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:55:25.478336 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.478335 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htmg4\" (UniqueName: \"kubernetes.io/projected/8c7975e2-bd3e-4007-87ca-f0875322ce62-kube-api-access-htmg4\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:55:25.478512 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.478346 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7975e2-bd3e-4007-87ca-f0875322ce62-tls-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:55:25.479535 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:25.479515 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/auth-test-model-44edfc-kserve-b675b9f47-22jzk"] Jul 24 00:55:26.002797 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:26.002763 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c7975e2-bd3e-4007-87ca-f0875322ce62" path="/var/lib/kubelet/pods/8c7975e2-bd3e-4007-87ca-f0875322ce62/volumes" Jul 24 00:55:26.188729 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:26.188691 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:55:36.593684 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.593593 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv"] Jul 24 00:55:36.594187 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.594118 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c7975e2-bd3e-4007-87ca-f0875322ce62" containerName="main" Jul 24 00:55:36.594187 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.594134 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c7975e2-bd3e-4007-87ca-f0875322ce62" containerName="main" Jul 24 00:55:36.594316 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.594198 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c7975e2-bd3e-4007-87ca-f0875322ce62" containerName="main" Jul 24 00:55:36.598563 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.598540 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.602123 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.602095 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-44edfc\"/\"kube-root-ca.crt\"" Jul 24 00:55:36.602253 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.602128 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-44edfc\"/\"openshift-service-ca.crt\"" Jul 24 00:55:36.602253 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.602146 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-44edfc\"/\"sub-test-model-44edfc-kserve-self-signed-certs\"" Jul 24 00:55:36.602253 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.602146 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-44edfc\"/\"default-dockercfg-848fn\"" Jul 24 00:55:36.608541 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.608516 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv"] Jul 24 00:55:36.686447 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.686407 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-tls-certs\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.686637 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.686472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv4xm\" (UniqueName: \"kubernetes.io/projected/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-kube-api-access-rv4xm\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.686637 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.686579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-model-cache\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.686637 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.686617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-home\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.686762 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.686685 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-dshm\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.686762 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.686705 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-tmp-dir\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.787345 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.787311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv4xm\" (UniqueName: \"kubernetes.io/projected/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-kube-api-access-rv4xm\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.787526 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.787376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-model-cache\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.787526 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.787405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-home\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.787526 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.787443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-dshm\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.787526 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.787468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-tmp-dir\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.787526 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.787488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-tls-certs\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.787882 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.787858 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-home\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.787972 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.787901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-model-cache\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.787972 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.787931 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-tmp-dir\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.789751 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.789733 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-dshm\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.790058 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.790041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-tls-certs\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.797349 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.797325 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv4xm\" (UniqueName: \"kubernetes.io/projected/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-kube-api-access-rv4xm\") pod \"sub-test-model-44edfc-kserve-797558fbb7-g6jzv\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:36.909499 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:36.909409 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:37.046251 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:37.046225 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv"] Jul 24 00:55:37.048268 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:55:37.048239 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7651a6ba_add3_4e5c_8bd5_b495c7677c7c.slice/crio-bcd2b5e8cd165bcea73e7a0bcedd7804df773a7a5b51e6fe04be84fcd255f78d WatchSource:0}: Error finding container bcd2b5e8cd165bcea73e7a0bcedd7804df773a7a5b51e6fe04be84fcd255f78d: Status 404 returned error can't find the container with id bcd2b5e8cd165bcea73e7a0bcedd7804df773a7a5b51e6fe04be84fcd255f78d Jul 24 00:55:37.522916 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:37.522880 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" event={"ID":"7651a6ba-add3-4e5c-8bd5-b495c7677c7c","Type":"ContainerStarted","Data":"6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7"} Jul 24 00:55:37.522916 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:37.522929 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" event={"ID":"7651a6ba-add3-4e5c-8bd5-b495c7677c7c","Type":"ContainerStarted","Data":"bcd2b5e8cd165bcea73e7a0bcedd7804df773a7a5b51e6fe04be84fcd255f78d"} Jul 24 00:55:37.541627 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:37.541573 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" podStartSLOduration=1.541557479 podStartE2EDuration="1.541557479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:55:37.540226459 +0000 UTC m=+2514.113323284" watchObservedRunningTime="2026-07-24 00:55:37.541557479 +0000 UTC m=+2514.114654279" Jul 24 00:55:39.178752 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:39.178715 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:55:46.910159 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:46.910115 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:46.910633 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:46.910177 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:46.922433 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:46.922409 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:47.596006 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:47.595977 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:55:59.391335 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:55:59.391293 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:56:17.286588 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:17.286550 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:56:23.102828 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:23.102785 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv"] Jul 24 00:56:34.991915 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:34.991874 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:56:38.122156 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.122089 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" podUID="7651a6ba-add3-4e5c-8bd5-b495c7677c7c" containerName="main" containerID="cri-o://6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7" gracePeriod=45 Jul 24 00:56:38.376795 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.376719 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:56:38.470631 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.470597 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-tmp-dir\") pod \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " Jul 24 00:56:38.470832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.470684 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-home\") pod \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " Jul 24 00:56:38.470832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.470707 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-dshm\") pod \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " Jul 24 00:56:38.470832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.470735 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-model-cache\") pod \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " Jul 24 00:56:38.470832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.470765 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-tls-certs\") pod \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " Jul 24 00:56:38.470832 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.470790 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv4xm\" (UniqueName: \"kubernetes.io/projected/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-kube-api-access-rv4xm\") pod \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\" (UID: \"7651a6ba-add3-4e5c-8bd5-b495c7677c7c\") " Jul 24 00:56:38.471127 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.470924 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7651a6ba-add3-4e5c-8bd5-b495c7677c7c" (UID: "7651a6ba-add3-4e5c-8bd5-b495c7677c7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:56:38.471127 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.470941 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-home" (OuterVolumeSpecName: "home") pod "7651a6ba-add3-4e5c-8bd5-b495c7677c7c" (UID: "7651a6ba-add3-4e5c-8bd5-b495c7677c7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:56:38.471127 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.471084 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7651a6ba-add3-4e5c-8bd5-b495c7677c7c" (UID: "7651a6ba-add3-4e5c-8bd5-b495c7677c7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:56:38.471295 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.471266 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-home\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:56:38.471295 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.471286 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-model-cache\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:56:38.471384 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.471304 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-tmp-dir\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:56:38.473082 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.473055 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-dshm" (OuterVolumeSpecName: "dshm") pod "7651a6ba-add3-4e5c-8bd5-b495c7677c7c" (UID: "7651a6ba-add3-4e5c-8bd5-b495c7677c7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:56:38.473203 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.473094 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-kube-api-access-rv4xm" (OuterVolumeSpecName: "kube-api-access-rv4xm") pod "7651a6ba-add3-4e5c-8bd5-b495c7677c7c" (UID: "7651a6ba-add3-4e5c-8bd5-b495c7677c7c"). InnerVolumeSpecName "kube-api-access-rv4xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:56:38.473203 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.473095 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7651a6ba-add3-4e5c-8bd5-b495c7677c7c" (UID: "7651a6ba-add3-4e5c-8bd5-b495c7677c7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:56:38.571862 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.571824 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-dshm\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:56:38.571862 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.571854 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-tls-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:56:38.571862 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.571867 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv4xm\" (UniqueName: \"kubernetes.io/projected/7651a6ba-add3-4e5c-8bd5-b495c7677c7c-kube-api-access-rv4xm\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:56:38.864702 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.864668 2576 generic.go:356] "Generic (PLEG): container finished" podID="7651a6ba-add3-4e5c-8bd5-b495c7677c7c" containerID="6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7" exitCode=0 Jul 24 00:56:38.864885 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.864713 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" Jul 24 00:56:38.864885 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.864751 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" event={"ID":"7651a6ba-add3-4e5c-8bd5-b495c7677c7c","Type":"ContainerDied","Data":"6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7"} Jul 24 00:56:38.864885 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.864796 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv" event={"ID":"7651a6ba-add3-4e5c-8bd5-b495c7677c7c","Type":"ContainerDied","Data":"bcd2b5e8cd165bcea73e7a0bcedd7804df773a7a5b51e6fe04be84fcd255f78d"} Jul 24 00:56:38.864885 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.864818 2576 scope.go:117] "RemoveContainer" containerID="6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7" Jul 24 00:56:38.875164 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.875141 2576 scope.go:117] "RemoveContainer" containerID="6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7" Jul 24 00:56:38.875471 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:56:38.875451 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7\": container with ID starting with 6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7 not found: ID does not exist" containerID="6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7" Jul 24 00:56:38.875523 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.875482 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7"} err="failed to get container status \"6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7\": rpc error: code = NotFound desc = could not find container \"6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7\": container with ID starting with 6cf535fb01582a60777b92f0205ef8843717bc3fdae4abd838d220ce2b804fd7 not found: ID does not exist" Jul 24 00:56:38.887059 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.887029 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv"] Jul 24 00:56:38.891287 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:38.891260 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/sub-test-model-44edfc-kserve-797558fbb7-g6jzv"] Jul 24 00:56:40.002119 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:56:40.002087 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7651a6ba-add3-4e5c-8bd5-b495c7677c7c" path="/var/lib/kubelet/pods/7651a6ba-add3-4e5c-8bd5-b495c7677c7c/volumes" Jul 24 00:57:05.699462 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.699420 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4"] Jul 24 00:57:05.699895 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.699816 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7651a6ba-add3-4e5c-8bd5-b495c7677c7c" containerName="main" Jul 24 00:57:05.699895 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.699827 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7651a6ba-add3-4e5c-8bd5-b495c7677c7c" containerName="main" Jul 24 00:57:05.699895 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.699883 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7651a6ba-add3-4e5c-8bd5-b495c7677c7c" containerName="main" Jul 24 00:57:05.702993 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.702976 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.705780 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.705754 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-44edfc\"/\"openshift-service-ca.crt\"" Jul 24 00:57:05.706591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.706575 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-44edfc\"/\"kube-root-ca.crt\"" Jul 24 00:57:05.706727 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.706702 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-44edfc\"/\"default-dockercfg-848fn\"" Jul 24 00:57:05.706727 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.706716 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-44edfc\"/\"rate-test-model-44edfc-kserve-self-signed-certs\"" Jul 24 00:57:05.710870 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.710602 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4"] Jul 24 00:57:05.824805 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.824762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-model-cache\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.825174 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.824834 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-dshm\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.825174 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.824877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a298cb1-9f85-48ef-a492-82b2d823c903-tls-certs\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.825174 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.824904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-tmp-dir\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.825174 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.825004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgpbk\" (UniqueName: \"kubernetes.io/projected/2a298cb1-9f85-48ef-a492-82b2d823c903-kube-api-access-bgpbk\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.825174 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.825101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-home\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.926075 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.926034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a298cb1-9f85-48ef-a492-82b2d823c903-tls-certs\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.926075 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.926078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-tmp-dir\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.926351 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.926104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgpbk\" (UniqueName: \"kubernetes.io/projected/2a298cb1-9f85-48ef-a492-82b2d823c903-kube-api-access-bgpbk\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.926351 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.926162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-home\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.926351 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.926188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-model-cache\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.926351 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.926246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-dshm\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.926574 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.926481 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-tmp-dir\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.926574 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.926540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-home\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.926668 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.926606 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-model-cache\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.928641 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.928612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-dshm\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.928928 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.928906 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a298cb1-9f85-48ef-a492-82b2d823c903-tls-certs\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:05.934388 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:05.934359 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgpbk\" (UniqueName: \"kubernetes.io/projected/2a298cb1-9f85-48ef-a492-82b2d823c903-kube-api-access-bgpbk\") pod \"rate-test-model-44edfc-kserve-68b7c96859-7vht4\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:06.013905 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:06.013803 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:06.160371 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:06.160326 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4"] Jul 24 00:57:06.162343 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:57:06.162300 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a298cb1_9f85_48ef_a492_82b2d823c903.slice/crio-2406e1a2563a6eb23c3124ce70a05c04c8d4cdd62886cd6e3daae2b5898884d9 WatchSource:0}: Error finding container 2406e1a2563a6eb23c3124ce70a05c04c8d4cdd62886cd6e3daae2b5898884d9: Status 404 returned error can't find the container with id 2406e1a2563a6eb23c3124ce70a05c04c8d4cdd62886cd6e3daae2b5898884d9 Jul 24 00:57:07.041188 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:07.041100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" event={"ID":"2a298cb1-9f85-48ef-a492-82b2d823c903","Type":"ContainerStarted","Data":"fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9"} Jul 24 00:57:07.041188 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:07.041155 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" event={"ID":"2a298cb1-9f85-48ef-a492-82b2d823c903","Type":"ContainerStarted","Data":"2406e1a2563a6eb23c3124ce70a05c04c8d4cdd62886cd6e3daae2b5898884d9"} Jul 24 00:57:07.058702 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:07.058644 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" podStartSLOduration=2.058624561 podStartE2EDuration="2.058624561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:57:07.055925083 +0000 UTC m=+2603.629021888" watchObservedRunningTime="2026-07-24 00:57:07.058624561 +0000 UTC m=+2603.631721355" Jul 24 00:57:08.983779 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:08.983727 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:57:16.014486 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:16.014454 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:16.014867 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:16.014508 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:16.027478 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:16.027442 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:16.102850 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:16.102815 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:57:33.094942 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:33.094901 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:57:47.895605 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:47.895567 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:57:55.801435 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:55.801398 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4"] Jul 24 00:57:56.821885 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:56.821846 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7358-140013700336176"] Jul 24 00:57:56.825244 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:56.825226 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013700336176" Jul 24 00:57:56.827786 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:56.827764 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wmrlv\"" Jul 24 00:57:56.831453 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:56.831428 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7358-140013700336176"] Jul 24 00:57:56.997723 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:56.997688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wvh6\" (UniqueName: \"kubernetes.io/projected/f8f071a3-430a-48f5-9bce-4eb58ce5b30e-kube-api-access-4wvh6\") pod \"test-curl-7358-140013700336176\" (UID: \"f8f071a3-430a-48f5-9bce-4eb58ce5b30e\") " pod="opendatahub/test-curl-7358-140013700336176" Jul 24 00:57:57.098481 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:57.098382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wvh6\" (UniqueName: \"kubernetes.io/projected/f8f071a3-430a-48f5-9bce-4eb58ce5b30e-kube-api-access-4wvh6\") pod \"test-curl-7358-140013700336176\" (UID: \"f8f071a3-430a-48f5-9bce-4eb58ce5b30e\") " pod="opendatahub/test-curl-7358-140013700336176" Jul 24 00:57:57.107266 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:57.107233 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wvh6\" (UniqueName: \"kubernetes.io/projected/f8f071a3-430a-48f5-9bce-4eb58ce5b30e-kube-api-access-4wvh6\") pod \"test-curl-7358-140013700336176\" (UID: \"f8f071a3-430a-48f5-9bce-4eb58ce5b30e\") " pod="opendatahub/test-curl-7358-140013700336176" Jul 24 00:57:57.135683 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:57.135632 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013700336176" Jul 24 00:57:57.487568 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:57.487536 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7358-140013700336176"] Jul 24 00:57:57.488716 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:57:57.488692 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8f071a3_430a_48f5_9bce_4eb58ce5b30e.slice/crio-648ed335b7e1c451e338dfbca2516e85076242bae8c187b36b544c3a32355389 WatchSource:0}: Error finding container 648ed335b7e1c451e338dfbca2516e85076242bae8c187b36b544c3a32355389: Status 404 returned error can't find the container with id 648ed335b7e1c451e338dfbca2516e85076242bae8c187b36b544c3a32355389 Jul 24 00:57:57.489984 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:57.489966 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 00:57:58.322081 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:58.322047 2576 generic.go:356] "Generic (PLEG): container finished" podID="f8f071a3-430a-48f5-9bce-4eb58ce5b30e" containerID="92149b45620c0d3c54b70578af370a1ebad8479893a3da468973622296b573ab" exitCode=0 Jul 24 00:57:58.322473 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:58.322133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7358-140013700336176" event={"ID":"f8f071a3-430a-48f5-9bce-4eb58ce5b30e","Type":"ContainerDied","Data":"92149b45620c0d3c54b70578af370a1ebad8479893a3da468973622296b573ab"} Jul 24 00:57:58.322473 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:58.322173 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7358-140013700336176" event={"ID":"f8f071a3-430a-48f5-9bce-4eb58ce5b30e","Type":"ContainerStarted","Data":"648ed335b7e1c451e338dfbca2516e85076242bae8c187b36b544c3a32355389"} Jul 24 00:57:59.473625 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.473592 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013700336176" Jul 24 00:57:59.516969 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.516913 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7358-140013700336176_f8f071a3-430a-48f5-9bce-4eb58ce5b30e/test-curl-7358-140013700336176/0.log" Jul 24 00:57:59.543318 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.543270 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7358-140013700336176"] Jul 24 00:57:59.546402 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.546372 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7358-140013700336176"] Jul 24 00:57:59.623050 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.622967 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wvh6\" (UniqueName: \"kubernetes.io/projected/f8f071a3-430a-48f5-9bce-4eb58ce5b30e-kube-api-access-4wvh6\") pod \"f8f071a3-430a-48f5-9bce-4eb58ce5b30e\" (UID: \"f8f071a3-430a-48f5-9bce-4eb58ce5b30e\") " Jul 24 00:57:59.625237 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.625202 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f071a3-430a-48f5-9bce-4eb58ce5b30e-kube-api-access-4wvh6" (OuterVolumeSpecName: "kube-api-access-4wvh6") pod "f8f071a3-430a-48f5-9bce-4eb58ce5b30e" (UID: "f8f071a3-430a-48f5-9bce-4eb58ce5b30e"). InnerVolumeSpecName "kube-api-access-4wvh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:57:59.672266 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.672232 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7358-140013700190384"] Jul 24 00:57:59.672601 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.672589 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f071a3-430a-48f5-9bce-4eb58ce5b30e" containerName="test-curl-7358-140013700336176" Jul 24 00:57:59.672644 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.672603 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f071a3-430a-48f5-9bce-4eb58ce5b30e" containerName="test-curl-7358-140013700336176" Jul 24 00:57:59.672694 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.672684 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f071a3-430a-48f5-9bce-4eb58ce5b30e" containerName="test-curl-7358-140013700336176" Jul 24 00:57:59.675796 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.675776 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013700190384" Jul 24 00:57:59.682505 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.682471 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7358-140013700190384"] Jul 24 00:57:59.724526 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.724489 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wvh6\" (UniqueName: \"kubernetes.io/projected/f8f071a3-430a-48f5-9bce-4eb58ce5b30e-kube-api-access-4wvh6\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:57:59.825620 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.825584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvlln\" (UniqueName: \"kubernetes.io/projected/86764965-917b-4b7a-9953-08ef2fc8db95-kube-api-access-xvlln\") pod \"test-curl-7358-140013700190384\" (UID: \"86764965-917b-4b7a-9953-08ef2fc8db95\") " pod="opendatahub/test-curl-7358-140013700190384" Jul 24 00:57:59.926715 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.926620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlln\" (UniqueName: \"kubernetes.io/projected/86764965-917b-4b7a-9953-08ef2fc8db95-kube-api-access-xvlln\") pod \"test-curl-7358-140013700190384\" (UID: \"86764965-917b-4b7a-9953-08ef2fc8db95\") " pod="opendatahub/test-curl-7358-140013700190384" Jul 24 00:57:59.934632 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.934610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlln\" (UniqueName: \"kubernetes.io/projected/86764965-917b-4b7a-9953-08ef2fc8db95-kube-api-access-xvlln\") pod \"test-curl-7358-140013700190384\" (UID: \"86764965-917b-4b7a-9953-08ef2fc8db95\") " pod="opendatahub/test-curl-7358-140013700190384" Jul 24 00:57:59.985625 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:57:59.985590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013700190384" Jul 24 00:58:00.003547 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:00.003512 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8f071a3-430a-48f5-9bce-4eb58ce5b30e" path="/var/lib/kubelet/pods/f8f071a3-430a-48f5-9bce-4eb58ce5b30e/volumes" Jul 24 00:58:00.123511 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:58:00.123474 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86764965_917b_4b7a_9953_08ef2fc8db95.slice/crio-967853cccd196f106e903b9b94968bc14c314b6a85f251be0aa348ff7b3cb947 WatchSource:0}: Error finding container 967853cccd196f106e903b9b94968bc14c314b6a85f251be0aa348ff7b3cb947: Status 404 returned error can't find the container with id 967853cccd196f106e903b9b94968bc14c314b6a85f251be0aa348ff7b3cb947 Jul 24 00:58:00.123985 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:00.123927 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7358-140013700190384"] Jul 24 00:58:00.334552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:00.334521 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013700336176" Jul 24 00:58:00.334732 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:00.334589 2576 scope.go:117] "RemoveContainer" containerID="92149b45620c0d3c54b70578af370a1ebad8479893a3da468973622296b573ab" Jul 24 00:58:00.336051 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:00.335841 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7358-140013700190384" event={"ID":"86764965-917b-4b7a-9953-08ef2fc8db95","Type":"ContainerStarted","Data":"967853cccd196f106e903b9b94968bc14c314b6a85f251be0aa348ff7b3cb947"} Jul 24 00:58:01.342624 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:01.342593 2576 generic.go:356] "Generic (PLEG): container finished" podID="86764965-917b-4b7a-9953-08ef2fc8db95" containerID="057dfb412a1fd4d6b118ca015970c2628502bd36206f1ddbe962517228f697b7" exitCode=0 Jul 24 00:58:01.343172 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:01.342651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7358-140013700190384" event={"ID":"86764965-917b-4b7a-9953-08ef2fc8db95","Type":"ContainerDied","Data":"057dfb412a1fd4d6b118ca015970c2628502bd36206f1ddbe962517228f697b7"} Jul 24 00:58:02.488437 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:02.488412 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013700190384" Jul 24 00:58:02.508855 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:02.508830 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7358-140013700190384_86764965-917b-4b7a-9953-08ef2fc8db95/test-curl-7358-140013700190384/0.log" Jul 24 00:58:02.534017 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:02.533988 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7358-140013700190384"] Jul 24 00:58:02.538103 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:02.538078 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7358-140013700190384"] Jul 24 00:58:02.549929 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:02.549909 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvlln\" (UniqueName: \"kubernetes.io/projected/86764965-917b-4b7a-9953-08ef2fc8db95-kube-api-access-xvlln\") pod \"86764965-917b-4b7a-9953-08ef2fc8db95\" (UID: \"86764965-917b-4b7a-9953-08ef2fc8db95\") " Jul 24 00:58:02.551910 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:02.551883 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86764965-917b-4b7a-9953-08ef2fc8db95-kube-api-access-xvlln" (OuterVolumeSpecName: "kube-api-access-xvlln") pod "86764965-917b-4b7a-9953-08ef2fc8db95" (UID: "86764965-917b-4b7a-9953-08ef2fc8db95"). InnerVolumeSpecName "kube-api-access-xvlln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:58:02.651235 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:02.651140 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvlln\" (UniqueName: \"kubernetes.io/projected/86764965-917b-4b7a-9953-08ef2fc8db95-kube-api-access-xvlln\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:58:03.358548 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:03.358519 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013700190384" Jul 24 00:58:03.358548 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:03.358538 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="967853cccd196f106e903b9b94968bc14c314b6a85f251be0aa348ff7b3cb947" Jul 24 00:58:03.368305 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:58:03.368275 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7358-140013700190384\" is forbidden: User \"system:node:ip-10-0-134-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-162.ec2.internal' and this object" podUID="86764965-917b-4b7a-9953-08ef2fc8db95" pod="opendatahub/test-curl-7358-140013700190384" Jul 24 00:58:04.000908 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:58:04.000863 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7358-140013700190384\" is forbidden: User \"system:node:ip-10-0-134-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-162.ec2.internal' and this object" podUID="86764965-917b-4b7a-9953-08ef2fc8db95" pod="opendatahub/test-curl-7358-140013700190384" Jul 24 00:58:04.002515 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:04.002492 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86764965-917b-4b7a-9953-08ef2fc8db95" path="/var/lib/kubelet/pods/86764965-917b-4b7a-9953-08ef2fc8db95/volumes" Jul 24 00:58:05.621500 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:05.621462 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7358-140013704328944"] Jul 24 00:58:05.621882 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:05.621871 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86764965-917b-4b7a-9953-08ef2fc8db95" containerName="test-curl-7358-140013700190384" Jul 24 00:58:05.621882 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:05.621883 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="86764965-917b-4b7a-9953-08ef2fc8db95" containerName="test-curl-7358-140013700190384" Jul 24 00:58:05.621994 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:05.621964 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="86764965-917b-4b7a-9953-08ef2fc8db95" containerName="test-curl-7358-140013700190384" Jul 24 00:58:05.625226 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:05.625209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013704328944" Jul 24 00:58:05.627589 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:05.627568 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wmrlv\"" Jul 24 00:58:05.632679 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:05.632651 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7358-140013704328944"] Jul 24 00:58:05.680534 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:05.680504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxksc\" (UniqueName: \"kubernetes.io/projected/d97f6110-8175-4508-a560-1c1d7bb7cc07-kube-api-access-rxksc\") pod \"test-curl-7358-140013704328944\" (UID: \"d97f6110-8175-4508-a560-1c1d7bb7cc07\") " pod="opendatahub/test-curl-7358-140013704328944" Jul 24 00:58:05.781927 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:05.781892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxksc\" (UniqueName: \"kubernetes.io/projected/d97f6110-8175-4508-a560-1c1d7bb7cc07-kube-api-access-rxksc\") pod \"test-curl-7358-140013704328944\" (UID: \"d97f6110-8175-4508-a560-1c1d7bb7cc07\") " pod="opendatahub/test-curl-7358-140013704328944" Jul 24 00:58:05.793023 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:05.791120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxksc\" (UniqueName: \"kubernetes.io/projected/d97f6110-8175-4508-a560-1c1d7bb7cc07-kube-api-access-rxksc\") pod \"test-curl-7358-140013704328944\" (UID: \"d97f6110-8175-4508-a560-1c1d7bb7cc07\") " pod="opendatahub/test-curl-7358-140013704328944" Jul 24 00:58:05.936482 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:05.936377 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013704328944" Jul 24 00:58:06.068941 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:06.068913 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7358-140013704328944"] Jul 24 00:58:06.070878 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:58:06.070849 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd97f6110_8175_4508_a560_1c1d7bb7cc07.slice/crio-da59b4e80e8ece02e0f06176ab633df4f591e140ee367da96ffe08a634accc24 WatchSource:0}: Error finding container da59b4e80e8ece02e0f06176ab633df4f591e140ee367da96ffe08a634accc24: Status 404 returned error can't find the container with id da59b4e80e8ece02e0f06176ab633df4f591e140ee367da96ffe08a634accc24 Jul 24 00:58:06.376666 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:06.376630 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7358-140013704328944" event={"ID":"d97f6110-8175-4508-a560-1c1d7bb7cc07","Type":"ContainerStarted","Data":"da59b4e80e8ece02e0f06176ab633df4f591e140ee367da96ffe08a634accc24"} Jul 24 00:58:06.688232 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:06.688201 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:58:07.384022 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:07.383984 2576 generic.go:356] "Generic (PLEG): container finished" podID="d97f6110-8175-4508-a560-1c1d7bb7cc07" containerID="1130a9689c188d53cb5d0db9231e9c8627ab29d8a5b4f227fc59412bc116e807" exitCode=0 Jul 24 00:58:07.384194 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:07.384069 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7358-140013704328944" event={"ID":"d97f6110-8175-4508-a560-1c1d7bb7cc07","Type":"ContainerDied","Data":"1130a9689c188d53cb5d0db9231e9c8627ab29d8a5b4f227fc59412bc116e807"} Jul 24 00:58:08.532696 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:08.532668 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013704328944" Jul 24 00:58:08.552116 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:08.552085 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7358-140013704328944_d97f6110-8175-4508-a560-1c1d7bb7cc07/test-curl-7358-140013704328944/0.log" Jul 24 00:58:08.578108 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:08.578071 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7358-140013704328944"] Jul 24 00:58:08.580610 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:08.580556 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7358-140013704328944"] Jul 24 00:58:08.605171 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:08.605137 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxksc\" (UniqueName: \"kubernetes.io/projected/d97f6110-8175-4508-a560-1c1d7bb7cc07-kube-api-access-rxksc\") pod \"d97f6110-8175-4508-a560-1c1d7bb7cc07\" (UID: \"d97f6110-8175-4508-a560-1c1d7bb7cc07\") " Jul 24 00:58:08.607318 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:08.607293 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97f6110-8175-4508-a560-1c1d7bb7cc07-kube-api-access-rxksc" (OuterVolumeSpecName: "kube-api-access-rxksc") pod "d97f6110-8175-4508-a560-1c1d7bb7cc07" (UID: "d97f6110-8175-4508-a560-1c1d7bb7cc07"). InnerVolumeSpecName "kube-api-access-rxksc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:58:08.706241 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:08.706145 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxksc\" (UniqueName: \"kubernetes.io/projected/d97f6110-8175-4508-a560-1c1d7bb7cc07-kube-api-access-rxksc\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:58:09.396865 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:09.396834 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013704328944" Jul 24 00:58:09.397054 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:09.396886 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da59b4e80e8ece02e0f06176ab633df4f591e140ee367da96ffe08a634accc24" Jul 24 00:58:09.406790 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:58:09.406762 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7358-140013704328944\" is forbidden: User \"system:node:ip-10-0-134-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-162.ec2.internal' and this object" podUID="d97f6110-8175-4508-a560-1c1d7bb7cc07" pod="opendatahub/test-curl-7358-140013704328944" Jul 24 00:58:10.002552 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:10.002508 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d97f6110-8175-4508-a560-1c1d7bb7cc07" path="/var/lib/kubelet/pods/d97f6110-8175-4508-a560-1c1d7bb7cc07/volumes" Jul 24 00:58:10.821152 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:10.821109 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" podUID="2a298cb1-9f85-48ef-a492-82b2d823c903" containerName="main" containerID="cri-o://fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9" gracePeriod=45 Jul 24 00:58:11.083811 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.083756 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:58:11.126397 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.126362 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-dshm\") pod \"2a298cb1-9f85-48ef-a492-82b2d823c903\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " Jul 24 00:58:11.126588 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.126427 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgpbk\" (UniqueName: \"kubernetes.io/projected/2a298cb1-9f85-48ef-a492-82b2d823c903-kube-api-access-bgpbk\") pod \"2a298cb1-9f85-48ef-a492-82b2d823c903\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " Jul 24 00:58:11.126588 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.126494 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-model-cache\") pod \"2a298cb1-9f85-48ef-a492-82b2d823c903\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " Jul 24 00:58:11.126588 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.126527 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a298cb1-9f85-48ef-a492-82b2d823c903-tls-certs\") pod \"2a298cb1-9f85-48ef-a492-82b2d823c903\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " Jul 24 00:58:11.126588 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.126558 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-home\") pod \"2a298cb1-9f85-48ef-a492-82b2d823c903\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " Jul 24 00:58:11.126588 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.126588 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-tmp-dir\") pod \"2a298cb1-9f85-48ef-a492-82b2d823c903\" (UID: \"2a298cb1-9f85-48ef-a492-82b2d823c903\") " Jul 24 00:58:11.126863 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.126774 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a298cb1-9f85-48ef-a492-82b2d823c903" (UID: "2a298cb1-9f85-48ef-a492-82b2d823c903"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:58:11.126920 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.126857 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-home" (OuterVolumeSpecName: "home") pod "2a298cb1-9f85-48ef-a492-82b2d823c903" (UID: "2a298cb1-9f85-48ef-a492-82b2d823c903"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:58:11.127032 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.127014 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-model-cache\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:58:11.127095 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.127036 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-home\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:58:11.127274 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.127249 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a298cb1-9f85-48ef-a492-82b2d823c903" (UID: "2a298cb1-9f85-48ef-a492-82b2d823c903"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:58:11.128643 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.128616 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-dshm" (OuterVolumeSpecName: "dshm") pod "2a298cb1-9f85-48ef-a492-82b2d823c903" (UID: "2a298cb1-9f85-48ef-a492-82b2d823c903"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:58:11.128643 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.128629 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a298cb1-9f85-48ef-a492-82b2d823c903-kube-api-access-bgpbk" (OuterVolumeSpecName: "kube-api-access-bgpbk") pod "2a298cb1-9f85-48ef-a492-82b2d823c903" (UID: "2a298cb1-9f85-48ef-a492-82b2d823c903"). InnerVolumeSpecName "kube-api-access-bgpbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:58:11.128785 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.128666 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a298cb1-9f85-48ef-a492-82b2d823c903-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a298cb1-9f85-48ef-a492-82b2d823c903" (UID: "2a298cb1-9f85-48ef-a492-82b2d823c903"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:58:11.228491 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.228452 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-dshm\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:58:11.228491 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.228485 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgpbk\" (UniqueName: \"kubernetes.io/projected/2a298cb1-9f85-48ef-a492-82b2d823c903-kube-api-access-bgpbk\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:58:11.228491 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.228495 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a298cb1-9f85-48ef-a492-82b2d823c903-tls-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:58:11.228741 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.228504 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a298cb1-9f85-48ef-a492-82b2d823c903-tmp-dir\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 00:58:11.409545 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.409454 2576 generic.go:356] "Generic (PLEG): container finished" podID="2a298cb1-9f85-48ef-a492-82b2d823c903" containerID="fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9" exitCode=0 Jul 24 00:58:11.409545 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.409515 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" Jul 24 00:58:11.409759 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.409542 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" event={"ID":"2a298cb1-9f85-48ef-a492-82b2d823c903","Type":"ContainerDied","Data":"fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9"} Jul 24 00:58:11.409759 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.409589 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4" event={"ID":"2a298cb1-9f85-48ef-a492-82b2d823c903","Type":"ContainerDied","Data":"2406e1a2563a6eb23c3124ce70a05c04c8d4cdd62886cd6e3daae2b5898884d9"} Jul 24 00:58:11.409759 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.409611 2576 scope.go:117] "RemoveContainer" containerID="fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9" Jul 24 00:58:11.419102 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.419083 2576 scope.go:117] "RemoveContainer" containerID="fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9" Jul 24 00:58:11.419350 ip-10-0-134-162 kubenswrapper[2576]: E0724 00:58:11.419329 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9\": container with ID starting with fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9 not found: ID does not exist" containerID="fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9" Jul 24 00:58:11.419429 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.419356 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9"} err="failed to get container status \"fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9\": rpc error: code = NotFound desc = could not find container \"fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9\": container with ID starting with fa170afe14c144544d03ed986a47fda0db317e4aa325f1dfd9d47d9fe4646ce9 not found: ID does not exist" Jul 24 00:58:11.430716 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.430690 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4"] Jul 24 00:58:11.433125 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:11.433104 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-44edfc/rate-test-model-44edfc-kserve-68b7c96859-7vht4"] Jul 24 00:58:12.004257 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:12.004225 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a298cb1-9f85-48ef-a492-82b2d823c903" path="/var/lib/kubelet/pods/2a298cb1-9f85-48ef-a492-82b2d823c903/volumes" Jul 24 00:58:15.379635 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.379600 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277"] Jul 24 00:58:15.380036 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.379994 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a298cb1-9f85-48ef-a492-82b2d823c903" containerName="main" Jul 24 00:58:15.380036 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.380005 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a298cb1-9f85-48ef-a492-82b2d823c903" containerName="main" Jul 24 00:58:15.380036 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.380014 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97f6110-8175-4508-a560-1c1d7bb7cc07" containerName="test-curl-7358-140013704328944" Jul 24 00:58:15.380036 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.380020 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97f6110-8175-4508-a560-1c1d7bb7cc07" containerName="test-curl-7358-140013704328944" Jul 24 00:58:15.380170 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.380076 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97f6110-8175-4508-a560-1c1d7bb7cc07" containerName="test-curl-7358-140013704328944" Jul 24 00:58:15.380170 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.380085 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a298cb1-9f85-48ef-a492-82b2d823c903" containerName="main" Jul 24 00:58:15.384410 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.384390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.386899 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.386873 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a3cf0e-openshift-default-dockercfg-nlmkn\"" Jul 24 00:58:15.394344 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.394321 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277"] Jul 24 00:58:15.469515 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.469479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-istio-data\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.469720 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.469542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bac8011-7aa9-407e-9249-47ade029c834-istio-podinfo\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.469720 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.469584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bac8011-7aa9-407e-9249-47ade029c834-istiod-ca-cert\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.469720 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.469639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-workload-certs\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.469720 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.469677 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glbq2\" (UniqueName: \"kubernetes.io/projected/6bac8011-7aa9-407e-9249-47ade029c834-kube-api-access-glbq2\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.469720 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.469719 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bac8011-7aa9-407e-9249-47ade029c834-istio-token\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.470022 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.469751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-workload-socket\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.470022 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.469802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-istio-envoy\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.470022 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.469883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-credential-socket\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.571377 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.571342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-workload-certs\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.571377 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.571378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glbq2\" (UniqueName: \"kubernetes.io/projected/6bac8011-7aa9-407e-9249-47ade029c834-kube-api-access-glbq2\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.571599 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.571408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bac8011-7aa9-407e-9249-47ade029c834-istio-token\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.571599 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.571515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-workload-socket\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.571599 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.571572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-istio-envoy\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.571759 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.571605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-credential-socket\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.571759 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.571631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-istio-data\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.571759 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.571665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bac8011-7aa9-407e-9249-47ade029c834-istio-podinfo\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.571759 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.571714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bac8011-7aa9-407e-9249-47ade029c834-istiod-ca-cert\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.571990 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.571768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-workload-certs\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.571990 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.571897 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-workload-socket\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.571990 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.571899 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-credential-socket\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.572091 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.572014 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-istio-data\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.572391 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.572375 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bac8011-7aa9-407e-9249-47ade029c834-istiod-ca-cert\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.573849 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.573830 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-istio-envoy\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.573934 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.573920 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bac8011-7aa9-407e-9249-47ade029c834-istio-podinfo\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.579931 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.579904 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bac8011-7aa9-407e-9249-47ade029c834-istio-token\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.580053 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.580002 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glbq2\" (UniqueName: \"kubernetes.io/projected/6bac8011-7aa9-407e-9249-47ade029c834-kube-api-access-glbq2\") pod \"e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.697029 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.696940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:15.852942 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.852910 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277"] Jul 24 00:58:15.854378 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.854341 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:58:15.854498 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.854421 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:58:15.854498 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:15.854471 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:58:16.440697 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:16.440656 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" event={"ID":"6bac8011-7aa9-407e-9249-47ade029c834","Type":"ContainerStarted","Data":"baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706"} Jul 24 00:58:16.441109 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:16.440723 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" event={"ID":"6bac8011-7aa9-407e-9249-47ade029c834","Type":"ContainerStarted","Data":"074abd7f411ede2b2d25b3b4db954871c0777be83a8f6a06d3c7faf2a6c15ba4"} Jul 24 00:58:16.462569 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:16.462514 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" podStartSLOduration=1.4624983999999999 podStartE2EDuration="1.4624984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:58:16.459787553 +0000 UTC m=+2673.032884357" watchObservedRunningTime="2026-07-24 00:58:16.4624984 +0000 UTC m=+2673.035595203" Jul 24 00:58:16.697189 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:16.697108 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:16.701801 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:16.701778 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:17.446222 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:17.446195 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:17.447304 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:17.447283 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:27.187764 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:27.187726 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:58:33.488153 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:33.488097 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 00:58:33.488695 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:33.488162 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" podUID="6bac8011-7aa9-407e-9249-47ade029c834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 00:58:35.547595 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:35.547552 2576 generic.go:356] "Generic (PLEG): container finished" podID="6bac8011-7aa9-407e-9249-47ade029c834" containerID="baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706" exitCode=0 Jul 24 00:58:35.548124 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:35.547600 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" event={"ID":"6bac8011-7aa9-407e-9249-47ade029c834","Type":"ContainerDied","Data":"baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706"} Jul 24 00:58:35.548124 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:35.548085 2576 scope.go:117] "RemoveContainer" containerID="baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706" Jul 24 00:58:35.549278 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:35.549251 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:58:35.549373 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:35.549344 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:58:35.549373 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:35.549375 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:58:36.560183 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:36.560076 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" event={"ID":"6bac8011-7aa9-407e-9249-47ade029c834","Type":"ContainerStarted","Data":"c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec"} Jul 24 00:58:36.697449 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:36.697412 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:36.697682 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:36.697661 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 24 00:58:36.697745 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:36.697710 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" podUID="6bac8011-7aa9-407e-9249-47ade029c834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 24 00:58:37.697517 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:37.697479 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 24 00:58:37.697913 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:37.697564 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" podUID="6bac8011-7aa9-407e-9249-47ade029c834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 24 00:58:38.698421 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:38.698383 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 24 00:58:38.698885 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:38.698445 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" podUID="6bac8011-7aa9-407e-9249-47ade029c834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 24 00:58:39.698189 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:39.698151 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 24 00:58:39.698398 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:39.698217 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" podUID="6bac8011-7aa9-407e-9249-47ade029c834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 24 00:58:40.700757 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:40.700726 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:40.701196 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:40.700989 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:40.701591 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:40.701572 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 00:58:44.177542 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.177515 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:58:44.184294 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.184268 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 00:58:44.452512 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.452422 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl"] Jul 24 00:58:44.457215 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.457195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.459920 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.459899 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-86b8da-openshift-default-dockercfg-qm97p\"" Jul 24 00:58:44.469771 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.469749 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl"] Jul 24 00:58:44.582155 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.582122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-token\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.582319 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.582169 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-workload-certs\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.582319 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.582228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istiod-ca-cert\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.582319 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.582262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-data\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.582319 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.582296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvvr\" (UniqueName: \"kubernetes.io/projected/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-kube-api-access-hxvvr\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.582477 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.582338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-workload-socket\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.582477 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.582392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-envoy\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.582557 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.582476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-credential-socket\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.582557 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.582512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-podinfo\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.683021 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.682984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvvr\" (UniqueName: \"kubernetes.io/projected/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-kube-api-access-hxvvr\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.683220 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.683042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-workload-socket\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.683220 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.683075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-envoy\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.683220 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.683138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-credential-socket\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.683220 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.683170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-podinfo\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.683220 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.683215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-token\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.683632 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.683278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-workload-certs\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.683632 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.683331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istiod-ca-cert\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.683632 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.683381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-data\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.683632 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.683545 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-workload-socket\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.683632 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.683610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-credential-socket\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.683912 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.683681 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-workload-certs\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.684191 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.684164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istiod-ca-cert\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.684306 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.684241 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-data\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.686054 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.686033 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-envoy\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.686278 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.686260 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-podinfo\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.690989 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.690969 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-token\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.691337 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.691318 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvvr\" (UniqueName: \"kubernetes.io/projected/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-kube-api-access-hxvvr\") pod \"e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.768500 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.768414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:44.901648 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.901619 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl"] Jul 24 00:58:44.904478 ip-10-0-134-162 kubenswrapper[2576]: W0724 00:58:44.904420 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c2b88b_9b1e_4bba_b7bb_0c285e292a34.slice/crio-33ac71b50e65553bccb41e26cfa4042eb69223371643a384c62fa0b065dfc317 WatchSource:0}: Error finding container 33ac71b50e65553bccb41e26cfa4042eb69223371643a384c62fa0b065dfc317: Status 404 returned error can't find the container with id 33ac71b50e65553bccb41e26cfa4042eb69223371643a384c62fa0b065dfc317 Jul 24 00:58:44.906733 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.906699 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:58:44.906843 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.906766 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:58:44.906843 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:44.906790 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:58:45.613362 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:45.613321 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" event={"ID":"55c2b88b-9b1e-4bba-b7bb-0c285e292a34","Type":"ContainerStarted","Data":"66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1"} Jul 24 00:58:45.613755 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:45.613385 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" event={"ID":"55c2b88b-9b1e-4bba-b7bb-0c285e292a34","Type":"ContainerStarted","Data":"33ac71b50e65553bccb41e26cfa4042eb69223371643a384c62fa0b065dfc317"} Jul 24 00:58:45.637160 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:45.637108 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" podStartSLOduration=1.637090376 podStartE2EDuration="1.637090376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:58:45.633670429 +0000 UTC m=+2702.206767243" watchObservedRunningTime="2026-07-24 00:58:45.637090376 +0000 UTC m=+2702.210187177" Jul 24 00:58:45.768931 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:45.768898 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:45.774036 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:45.774014 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:46.618994 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:46.618965 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:46.620070 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:46.620044 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:58:57.185959 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:58:57.185907 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:59:04.724586 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:04.724551 2576 generic.go:356] "Generic (PLEG): container finished" podID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerID="66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1" exitCode=0 Jul 24 00:59:04.725047 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:04.724635 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" event={"ID":"55c2b88b-9b1e-4bba-b7bb-0c285e292a34","Type":"ContainerDied","Data":"66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1"} Jul 24 00:59:04.725141 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:04.725116 2576 scope.go:117] "RemoveContainer" containerID="66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1" Jul 24 00:59:04.726197 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:04.726168 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:59:04.726281 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:04.726234 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:59:04.726281 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:04.726262 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:59:05.731984 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:05.731926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" event={"ID":"55c2b88b-9b1e-4bba-b7bb-0c285e292a34","Type":"ContainerStarted","Data":"0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac"} Jul 24 00:59:05.769106 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:05.769075 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:59:05.769266 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:05.769228 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 24 00:59:05.769310 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:05.769270 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 24 00:59:06.769780 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:06.769742 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 24 00:59:06.770190 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:06.769827 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 24 00:59:07.769642 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:07.769611 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 24 00:59:07.769820 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:07.769666 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 24 00:59:08.769449 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:08.769415 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 24 00:59:08.769620 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:08.769512 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 24 00:59:09.769645 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:09.769613 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 24 00:59:09.770068 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:09.769663 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 24 00:59:10.773170 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:10.773133 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:59:10.773549 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:10.773418 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:59:10.774295 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:10.774278 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 00:59:43.189466 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:43.189432 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 00:59:54.886465 ip-10-0-134-162 kubenswrapper[2576]: I0724 00:59:54.886429 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 01:00:00.142204 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:00.142167 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck"] Jul 24 01:00:00.145746 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:00.145725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck" Jul 24 01:00:00.149636 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:00.149615 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\"" Jul 24 01:00:00.154912 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:00.154641 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck"] Jul 24 01:00:00.159401 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:00.159379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck" Jul 24 01:00:00.506444 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:00.506416 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck"] Jul 24 01:00:00.507819 ip-10-0-134-162 kubenswrapper[2576]: W0724 01:00:00.507788 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f75243_86a9_451f_95c2_534845c2c1d0.slice/crio-8a0de7de16e492e6b9c6afbaa15ef7aacd5693f58cc39d244c4feda8507f9243 WatchSource:0}: Error finding container 8a0de7de16e492e6b9c6afbaa15ef7aacd5693f58cc39d244c4feda8507f9243: Status 404 returned error can't find the container with id 8a0de7de16e492e6b9c6afbaa15ef7aacd5693f58cc39d244c4feda8507f9243 Jul 24 01:00:01.057438 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:01.057401 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck" event={"ID":"94f75243-86a9-451f-95c2-534845c2c1d0","Type":"ContainerStarted","Data":"4413c46015d52714a804e17ce72fd9d5966c21e84c117b0954fc5bc70f0a0843"} Jul 24 01:00:01.057621 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:01.057454 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck" event={"ID":"94f75243-86a9-451f-95c2-534845c2c1d0","Type":"ContainerStarted","Data":"8a0de7de16e492e6b9c6afbaa15ef7aacd5693f58cc39d244c4feda8507f9243"} Jul 24 01:00:01.073609 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:01.073551 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck" podStartSLOduration=1.073533442 podStartE2EDuration="1.073533442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:00:01.072015882 +0000 UTC m=+2777.645112686" watchObservedRunningTime="2026-07-24 01:00:01.073533442 +0000 UTC m=+2777.646630239" Jul 24 01:00:02.064355 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:02.064322 2576 generic.go:356] "Generic (PLEG): container finished" podID="94f75243-86a9-451f-95c2-534845c2c1d0" containerID="4413c46015d52714a804e17ce72fd9d5966c21e84c117b0954fc5bc70f0a0843" exitCode=0 Jul 24 01:00:02.064760 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:02.064405 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck" event={"ID":"94f75243-86a9-451f-95c2-534845c2c1d0","Type":"ContainerDied","Data":"4413c46015d52714a804e17ce72fd9d5966c21e84c117b0954fc5bc70f0a0843"} Jul 24 01:00:03.218704 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:03.218676 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck" Jul 24 01:00:04.076929 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:04.076896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck" event={"ID":"94f75243-86a9-451f-95c2-534845c2c1d0","Type":"ContainerDied","Data":"8a0de7de16e492e6b9c6afbaa15ef7aacd5693f58cc39d244c4feda8507f9243"} Jul 24 01:00:04.077154 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:04.076934 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck" Jul 24 01:00:04.077154 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:04.076941 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a0de7de16e492e6b9c6afbaa15ef7aacd5693f58cc39d244c4feda8507f9243" Jul 24 01:00:05.444920 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.444886 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f"] Jul 24 01:00:05.445518 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.445497 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f75243-86a9-451f-95c2-534845c2c1d0" containerName="cleanup" Jul 24 01:00:05.445603 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.445520 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f75243-86a9-451f-95c2-534845c2c1d0" containerName="cleanup" Jul 24 01:00:05.445661 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.445620 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f75243-86a9-451f-95c2-534845c2c1d0" containerName="cleanup" Jul 24 01:00:05.449256 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.449231 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.451855 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.451835 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-e040d3-openshift-default-dockercfg-kcgm2\"" Jul 24 01:00:05.460274 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.460222 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f"] Jul 24 01:00:05.589316 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.589286 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-credential-socket\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.589526 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.589327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-workload-certs\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.589526 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.589349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-token\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.589526 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.589439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2021f4-39cd-4772-8a57-f92d11e297f4-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.589526 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.589488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-podinfo\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.589756 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.589532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-envoy\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.589756 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.589578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz4zm\" (UniqueName: \"kubernetes.io/projected/0f2021f4-39cd-4772-8a57-f92d11e297f4-kube-api-access-dz4zm\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.589756 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.589663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-workload-socket\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.589756 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.589713 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-data\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.690344 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.690309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-workload-certs\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.690344 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.690346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-token\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.690616 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.690370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2021f4-39cd-4772-8a57-f92d11e297f4-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.690616 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.690489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-podinfo\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.690616 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.690535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-envoy\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.690790 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.690661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz4zm\" (UniqueName: \"kubernetes.io/projected/0f2021f4-39cd-4772-8a57-f92d11e297f4-kube-api-access-dz4zm\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.690790 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.690753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-workload-socket\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.690790 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.690781 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-workload-certs\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.690998 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.690818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-data\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.690998 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.690877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-credential-socket\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.691121 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.691106 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-workload-socket\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.691238 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.691215 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-data\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.691289 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.691263 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2021f4-39cd-4772-8a57-f92d11e297f4-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.691325 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.691289 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-credential-socket\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.692807 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.692787 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-envoy\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.693137 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.693119 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-podinfo\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.698654 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.698598 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-token\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.698874 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.698853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz4zm\" (UniqueName: \"kubernetes.io/projected/0f2021f4-39cd-4772-8a57-f92d11e297f4-kube-api-access-dz4zm\") pod \"e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.761991 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.761930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:05.903500 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.903473 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f"] Jul 24 01:00:05.904313 ip-10-0-134-162 kubenswrapper[2576]: W0724 01:00:05.904289 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2021f4_39cd_4772_8a57_f92d11e297f4.slice/crio-0cb95f90a74465c556fde007f672f6086e7a6dbad66e335c24ab0e716bfb8678 WatchSource:0}: Error finding container 0cb95f90a74465c556fde007f672f6086e7a6dbad66e335c24ab0e716bfb8678: Status 404 returned error can't find the container with id 0cb95f90a74465c556fde007f672f6086e7a6dbad66e335c24ab0e716bfb8678 Jul 24 01:00:05.906495 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.906464 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 01:00:05.906595 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.906530 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 01:00:05.906595 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:05.906562 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 01:00:06.089899 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:06.089864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" event={"ID":"0f2021f4-39cd-4772-8a57-f92d11e297f4","Type":"ContainerStarted","Data":"8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f"} Jul 24 01:00:06.090113 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:06.089919 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" event={"ID":"0f2021f4-39cd-4772-8a57-f92d11e297f4","Type":"ContainerStarted","Data":"0cb95f90a74465c556fde007f672f6086e7a6dbad66e335c24ab0e716bfb8678"} Jul 24 01:00:06.126230 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:06.126169 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" podStartSLOduration=1.126152058 podStartE2EDuration="1.126152058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:00:06.122611436 +0000 UTC m=+2782.695708253" watchObservedRunningTime="2026-07-24 01:00:06.126152058 +0000 UTC m=+2782.699248860" Jul 24 01:00:06.762608 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:06.762529 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:06.768832 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:06.768802 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:07.096011 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:07.095977 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:07.096923 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:07.096897 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:12.686065 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:12.686031 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 01:00:21.189856 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:21.189825 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerID="8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f" exitCode=0 Jul 24 01:00:21.190323 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:21.189896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" event={"ID":"0f2021f4-39cd-4772-8a57-f92d11e297f4","Type":"ContainerDied","Data":"8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f"} Jul 24 01:00:21.190323 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:21.190315 2576 scope.go:117] "RemoveContainer" containerID="8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f" Jul 24 01:00:21.191467 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:21.191437 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 01:00:21.191528 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:21.191506 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 01:00:21.191569 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:21.191540 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 01:00:22.096916 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:22.096877 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:22.197962 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:22.197910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" event={"ID":"0f2021f4-39cd-4772-8a57-f92d11e297f4","Type":"ContainerStarted","Data":"def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44"} Jul 24 01:00:22.763097 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:22.763055 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:22.764321 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:22.764291 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Jul 24 01:00:22.764458 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:22.764362 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" podUID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Jul 24 01:00:23.762697 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:23.762652 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Jul 24 01:00:23.763098 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:23.762739 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" podUID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Jul 24 01:00:24.762348 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:24.762311 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Jul 24 01:00:24.762529 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:24.762370 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" podUID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Jul 24 01:00:25.762348 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:25.762313 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Jul 24 01:00:25.762781 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:25.762369 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" podUID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Jul 24 01:00:26.766596 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:26.766564 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:26.767059 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:26.766983 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:26.767730 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:26.767707 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:38.289753 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:38.289722 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 01:00:45.090515 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:45.090474 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f"] Jul 24 01:00:45.091007 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:45.090774 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" podUID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerName="istio-proxy" containerID="cri-o://def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44" gracePeriod=30 Jul 24 01:00:50.349159 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.349133 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:50.371245 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.371147 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerID="def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44" exitCode=0 Jul 24 01:00:50.371245 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.371195 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" event={"ID":"0f2021f4-39cd-4772-8a57-f92d11e297f4","Type":"ContainerDied","Data":"def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44"} Jul 24 01:00:50.371484 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.371213 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" Jul 24 01:00:50.371484 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.371292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f" event={"ID":"0f2021f4-39cd-4772-8a57-f92d11e297f4","Type":"ContainerDied","Data":"0cb95f90a74465c556fde007f672f6086e7a6dbad66e335c24ab0e716bfb8678"} Jul 24 01:00:50.371484 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.371322 2576 scope.go:117] "RemoveContainer" containerID="def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44" Jul 24 01:00:50.382386 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.382362 2576 scope.go:117] "RemoveContainer" containerID="8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f" Jul 24 01:00:50.391422 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.391396 2576 scope.go:117] "RemoveContainer" containerID="def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44" Jul 24 01:00:50.391743 ip-10-0-134-162 kubenswrapper[2576]: E0724 01:00:50.391713 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44\": container with ID starting with def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44 not found: ID does not exist" containerID="def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44" Jul 24 01:00:50.391849 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.391750 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44"} err="failed to get container status \"def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44\": rpc error: code = NotFound desc = could not find container \"def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44\": container with ID starting with def11cbaa21e65db0f3bb26227be569ef2d084a1e8cb993cc3eaa041e7938d44 not found: ID does not exist" Jul 24 01:00:50.391849 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.391770 2576 scope.go:117] "RemoveContainer" containerID="8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f" Jul 24 01:00:50.392059 ip-10-0-134-162 kubenswrapper[2576]: E0724 01:00:50.392035 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f\": container with ID starting with 8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f not found: ID does not exist" containerID="8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f" Jul 24 01:00:50.392108 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.392069 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f"} err="failed to get container status \"8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f\": rpc error: code = NotFound desc = could not find container \"8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f\": container with ID starting with 8142d8a35963268bee130b43ebdca72cbda671321c1132142da366c632c41c0f not found: ID does not exist" Jul 24 01:00:50.446594 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.446536 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-workload-certs\") pod \"0f2021f4-39cd-4772-8a57-f92d11e297f4\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " Jul 24 01:00:50.446594 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.446593 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-data\") pod \"0f2021f4-39cd-4772-8a57-f92d11e297f4\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " Jul 24 01:00:50.446858 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.446621 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz4zm\" (UniqueName: \"kubernetes.io/projected/0f2021f4-39cd-4772-8a57-f92d11e297f4-kube-api-access-dz4zm\") pod \"0f2021f4-39cd-4772-8a57-f92d11e297f4\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " Jul 24 01:00:50.446858 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.446672 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2021f4-39cd-4772-8a57-f92d11e297f4-istiod-ca-cert\") pod \"0f2021f4-39cd-4772-8a57-f92d11e297f4\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " Jul 24 01:00:50.446858 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.446721 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-workload-socket\") pod \"0f2021f4-39cd-4772-8a57-f92d11e297f4\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " Jul 24 01:00:50.446858 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.446753 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-credential-socket\") pod \"0f2021f4-39cd-4772-8a57-f92d11e297f4\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " Jul 24 01:00:50.446858 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.446791 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-token\") pod \"0f2021f4-39cd-4772-8a57-f92d11e297f4\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " Jul 24 01:00:50.447209 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.446889 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-podinfo\") pod \"0f2021f4-39cd-4772-8a57-f92d11e297f4\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " Jul 24 01:00:50.447209 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.446973 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-envoy\") pod \"0f2021f4-39cd-4772-8a57-f92d11e297f4\" (UID: \"0f2021f4-39cd-4772-8a57-f92d11e297f4\") " Jul 24 01:00:50.447209 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.446934 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f2021f4-39cd-4772-8a57-f92d11e297f4" (UID: "0f2021f4-39cd-4772-8a57-f92d11e297f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:00:50.447390 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.447203 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f2021f4-39cd-4772-8a57-f92d11e297f4" (UID: "0f2021f4-39cd-4772-8a57-f92d11e297f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:00:50.447390 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.447301 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2021f4-39cd-4772-8a57-f92d11e297f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f2021f4-39cd-4772-8a57-f92d11e297f4" (UID: "0f2021f4-39cd-4772-8a57-f92d11e297f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 01:00:50.447589 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.447566 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f2021f4-39cd-4772-8a57-f92d11e297f4" (UID: "0f2021f4-39cd-4772-8a57-f92d11e297f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:00:50.447853 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.447826 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f2021f4-39cd-4772-8a57-f92d11e297f4" (UID: "0f2021f4-39cd-4772-8a57-f92d11e297f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:00:50.449168 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.449144 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f2021f4-39cd-4772-8a57-f92d11e297f4" (UID: "0f2021f4-39cd-4772-8a57-f92d11e297f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 01:00:50.449168 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.449153 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2021f4-39cd-4772-8a57-f92d11e297f4-kube-api-access-dz4zm" (OuterVolumeSpecName: "kube-api-access-dz4zm") pod "0f2021f4-39cd-4772-8a57-f92d11e297f4" (UID: "0f2021f4-39cd-4772-8a57-f92d11e297f4"). InnerVolumeSpecName "kube-api-access-dz4zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 01:00:50.449301 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.449172 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f2021f4-39cd-4772-8a57-f92d11e297f4" (UID: "0f2021f4-39cd-4772-8a57-f92d11e297f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:00:50.449301 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.449205 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f2021f4-39cd-4772-8a57-f92d11e297f4" (UID: "0f2021f4-39cd-4772-8a57-f92d11e297f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 01:00:50.548720 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.548674 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:00:50.548720 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.548712 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:00:50.548720 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.548721 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz4zm\" (UniqueName: \"kubernetes.io/projected/0f2021f4-39cd-4772-8a57-f92d11e297f4-kube-api-access-dz4zm\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:00:50.548720 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.548732 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2021f4-39cd-4772-8a57-f92d11e297f4-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:00:50.549043 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.548741 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:00:50.549043 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.548749 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:00:50.549043 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.548756 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:00:50.549043 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.548766 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:00:50.549043 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.548778 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2021f4-39cd-4772-8a57-f92d11e297f4-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:00:50.695905 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.695867 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f"] Jul 24 01:00:50.700184 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:50.700152 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e040d3-openshift-default-84f678d99b-md57f"] Jul 24 01:00:52.002898 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:00:52.002861 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f2021f4-39cd-4772-8a57-f92d11e297f4" path="/var/lib/kubelet/pods/0f2021f4-39cd-4772-8a57-f92d11e297f4/volumes" Jul 24 01:01:03.493498 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:01:03.493453 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 01:01:14.388536 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:01:14.388493 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 01:01:57.418802 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:01:57.418756 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277"] Jul 24 01:01:57.419385 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:01:57.419047 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" podUID="6bac8011-7aa9-407e-9249-47ade029c834" containerName="istio-proxy" containerID="cri-o://c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec" gracePeriod=30 Jul 24 01:02:02.447051 ip-10-0-134-162 kubenswrapper[2576]: E0724 01:02:02.447020 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bac8011_7aa9_407e_9249_47ade029c834.slice/crio-conmon-c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec.scope\": RecentStats: unable to find data in memory cache]" Jul 24 01:02:02.681295 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.681231 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 01:02:02.800773 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.800739 2576 generic.go:356] "Generic (PLEG): container finished" podID="6bac8011-7aa9-407e-9249-47ade029c834" containerID="c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec" exitCode=0 Jul 24 01:02:02.800999 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.800786 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" Jul 24 01:02:02.800999 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.800840 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" event={"ID":"6bac8011-7aa9-407e-9249-47ade029c834","Type":"ContainerDied","Data":"c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec"} Jul 24 01:02:02.800999 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.800901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277" event={"ID":"6bac8011-7aa9-407e-9249-47ade029c834","Type":"ContainerDied","Data":"074abd7f411ede2b2d25b3b4db954871c0777be83a8f6a06d3c7faf2a6c15ba4"} Jul 24 01:02:02.800999 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.800932 2576 scope.go:117] "RemoveContainer" containerID="c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec" Jul 24 01:02:02.810385 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.810370 2576 scope.go:117] "RemoveContainer" containerID="baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706" Jul 24 01:02:02.818608 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.818591 2576 scope.go:117] "RemoveContainer" containerID="c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec" Jul 24 01:02:02.818869 ip-10-0-134-162 kubenswrapper[2576]: E0724 01:02:02.818837 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec\": container with ID starting with c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec not found: ID does not exist" containerID="c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec" Jul 24 01:02:02.818940 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.818876 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec"} err="failed to get container status \"c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec\": rpc error: code = NotFound desc = could not find container \"c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec\": container with ID starting with c3a4235892b53cef6ba1719388a075b8464c23c918a598b2a10caa78526b62ec not found: ID does not exist" Jul 24 01:02:02.818940 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.818898 2576 scope.go:117] "RemoveContainer" containerID="baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706" Jul 24 01:02:02.819185 ip-10-0-134-162 kubenswrapper[2576]: E0724 01:02:02.819165 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706\": container with ID starting with baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706 not found: ID does not exist" containerID="baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706" Jul 24 01:02:02.819244 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.819195 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706"} err="failed to get container status \"baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706\": rpc error: code = NotFound desc = could not find container \"baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706\": container with ID starting with baf2f12b8d60c9a546830f3cc88476962b346105c1015f056aaa3872216d0706 not found: ID does not exist" Jul 24 01:02:02.852697 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.852674 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-istio-data\") pod \"6bac8011-7aa9-407e-9249-47ade029c834\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " Jul 24 01:02:02.852775 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.852701 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glbq2\" (UniqueName: \"kubernetes.io/projected/6bac8011-7aa9-407e-9249-47ade029c834-kube-api-access-glbq2\") pod \"6bac8011-7aa9-407e-9249-47ade029c834\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " Jul 24 01:02:02.852775 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.852764 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bac8011-7aa9-407e-9249-47ade029c834-istiod-ca-cert\") pod \"6bac8011-7aa9-407e-9249-47ade029c834\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " Jul 24 01:02:02.852918 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.852798 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-istio-envoy\") pod \"6bac8011-7aa9-407e-9249-47ade029c834\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " Jul 24 01:02:02.852918 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.852818 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-workload-certs\") pod \"6bac8011-7aa9-407e-9249-47ade029c834\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " Jul 24 01:02:02.852918 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.852834 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bac8011-7aa9-407e-9249-47ade029c834-istio-token\") pod \"6bac8011-7aa9-407e-9249-47ade029c834\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " Jul 24 01:02:02.852918 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.852853 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-credential-socket\") pod \"6bac8011-7aa9-407e-9249-47ade029c834\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " Jul 24 01:02:02.852918 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.852892 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-workload-socket\") pod \"6bac8011-7aa9-407e-9249-47ade029c834\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " Jul 24 01:02:02.853187 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.852983 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bac8011-7aa9-407e-9249-47ade029c834-istio-podinfo\") pod \"6bac8011-7aa9-407e-9249-47ade029c834\" (UID: \"6bac8011-7aa9-407e-9249-47ade029c834\") " Jul 24 01:02:02.853187 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.853167 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6bac8011-7aa9-407e-9249-47ade029c834" (UID: "6bac8011-7aa9-407e-9249-47ade029c834"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:02.853389 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.853316 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bac8011-7aa9-407e-9249-47ade029c834-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6bac8011-7aa9-407e-9249-47ade029c834" (UID: "6bac8011-7aa9-407e-9249-47ade029c834"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 01:02:02.853389 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.853338 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:02.853519 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.853461 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6bac8011-7aa9-407e-9249-47ade029c834" (UID: "6bac8011-7aa9-407e-9249-47ade029c834"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:02.853519 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.853476 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6bac8011-7aa9-407e-9249-47ade029c834" (UID: "6bac8011-7aa9-407e-9249-47ade029c834"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:02.853605 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.853551 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-istio-data" (OuterVolumeSpecName: "istio-data") pod "6bac8011-7aa9-407e-9249-47ade029c834" (UID: "6bac8011-7aa9-407e-9249-47ade029c834"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:02.855131 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.855076 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bac8011-7aa9-407e-9249-47ade029c834-kube-api-access-glbq2" (OuterVolumeSpecName: "kube-api-access-glbq2") pod "6bac8011-7aa9-407e-9249-47ade029c834" (UID: "6bac8011-7aa9-407e-9249-47ade029c834"). InnerVolumeSpecName "kube-api-access-glbq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 01:02:02.855131 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.855099 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6bac8011-7aa9-407e-9249-47ade029c834" (UID: "6bac8011-7aa9-407e-9249-47ade029c834"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:02.855459 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.855444 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bac8011-7aa9-407e-9249-47ade029c834-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6bac8011-7aa9-407e-9249-47ade029c834" (UID: "6bac8011-7aa9-407e-9249-47ade029c834"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 01:02:02.855537 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.855504 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bac8011-7aa9-407e-9249-47ade029c834-istio-token" (OuterVolumeSpecName: "istio-token") pod "6bac8011-7aa9-407e-9249-47ade029c834" (UID: "6bac8011-7aa9-407e-9249-47ade029c834"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 01:02:02.954057 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.953974 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bac8011-7aa9-407e-9249-47ade029c834-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:02.954057 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.954002 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:02.954057 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.954012 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:02.954057 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.954020 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bac8011-7aa9-407e-9249-47ade029c834-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:02.954057 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.954028 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:02.954057 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.954037 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glbq2\" (UniqueName: \"kubernetes.io/projected/6bac8011-7aa9-407e-9249-47ade029c834-kube-api-access-glbq2\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:02.954057 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.954046 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bac8011-7aa9-407e-9249-47ade029c834-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:02.954057 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:02.954053 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bac8011-7aa9-407e-9249-47ade029c834-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:03.125739 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:03.125708 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277"] Jul 24 01:02:03.128068 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:03.128048 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3cf0e-openshift-default-764f948466-kp277"] Jul 24 01:02:04.001620 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:04.001588 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bac8011-7aa9-407e-9249-47ade029c834" path="/var/lib/kubelet/pods/6bac8011-7aa9-407e-9249-47ade029c834/volumes" Jul 24 01:02:11.489789 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:11.489745 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 01:02:19.507272 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:19.507181 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl"] Jul 24 01:02:19.507694 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:19.507388 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerName="istio-proxy" containerID="cri-o://0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac" gracePeriod=30 Jul 24 01:02:24.772766 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.772703 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 01:02:24.861024 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.860993 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-token\") pod \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " Jul 24 01:02:24.861204 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861034 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istiod-ca-cert\") pod \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " Jul 24 01:02:24.861204 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861062 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-workload-certs\") pod \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " Jul 24 01:02:24.861204 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861083 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-credential-socket\") pod \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " Jul 24 01:02:24.861204 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861113 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxvvr\" (UniqueName: \"kubernetes.io/projected/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-kube-api-access-hxvvr\") pod \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " Jul 24 01:02:24.861204 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861146 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-podinfo\") pod \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " Jul 24 01:02:24.861204 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861195 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-data\") pod \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " Jul 24 01:02:24.861529 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861237 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-workload-socket\") pod \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " Jul 24 01:02:24.861529 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861313 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-envoy\") pod \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\" (UID: \"55c2b88b-9b1e-4bba-b7bb-0c285e292a34\") " Jul 24 01:02:24.861529 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861414 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "55c2b88b-9b1e-4bba-b7bb-0c285e292a34" (UID: "55c2b88b-9b1e-4bba-b7bb-0c285e292a34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 01:02:24.861529 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861510 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "55c2b88b-9b1e-4bba-b7bb-0c285e292a34" (UID: "55c2b88b-9b1e-4bba-b7bb-0c285e292a34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:24.861743 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861707 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:24.861743 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861735 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:24.861854 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861708 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "55c2b88b-9b1e-4bba-b7bb-0c285e292a34" (UID: "55c2b88b-9b1e-4bba-b7bb-0c285e292a34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:24.861854 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.861749 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "55c2b88b-9b1e-4bba-b7bb-0c285e292a34" (UID: "55c2b88b-9b1e-4bba-b7bb-0c285e292a34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:24.862096 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.862074 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-data" (OuterVolumeSpecName: "istio-data") pod "55c2b88b-9b1e-4bba-b7bb-0c285e292a34" (UID: "55c2b88b-9b1e-4bba-b7bb-0c285e292a34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:24.863460 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.863432 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "55c2b88b-9b1e-4bba-b7bb-0c285e292a34" (UID: "55c2b88b-9b1e-4bba-b7bb-0c285e292a34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:24.863558 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.863474 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "55c2b88b-9b1e-4bba-b7bb-0c285e292a34" (UID: "55c2b88b-9b1e-4bba-b7bb-0c285e292a34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 01:02:24.863717 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.863700 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-token" (OuterVolumeSpecName: "istio-token") pod "55c2b88b-9b1e-4bba-b7bb-0c285e292a34" (UID: "55c2b88b-9b1e-4bba-b7bb-0c285e292a34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 01:02:24.863775 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.863712 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-kube-api-access-hxvvr" (OuterVolumeSpecName: "kube-api-access-hxvvr") pod "55c2b88b-9b1e-4bba-b7bb-0c285e292a34" (UID: "55c2b88b-9b1e-4bba-b7bb-0c285e292a34"). InnerVolumeSpecName "kube-api-access-hxvvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 01:02:24.937446 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.937409 2576 generic.go:356] "Generic (PLEG): container finished" podID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerID="0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac" exitCode=0 Jul 24 01:02:24.937446 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.937448 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" event={"ID":"55c2b88b-9b1e-4bba-b7bb-0c285e292a34","Type":"ContainerDied","Data":"0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac"} Jul 24 01:02:24.937671 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.937459 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" Jul 24 01:02:24.937671 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.937493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl" event={"ID":"55c2b88b-9b1e-4bba-b7bb-0c285e292a34","Type":"ContainerDied","Data":"33ac71b50e65553bccb41e26cfa4042eb69223371643a384c62fa0b065dfc317"} Jul 24 01:02:24.937671 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.937515 2576 scope.go:117] "RemoveContainer" containerID="0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac" Jul 24 01:02:24.947412 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.947394 2576 scope.go:117] "RemoveContainer" containerID="66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1" Jul 24 01:02:24.955571 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.955554 2576 scope.go:117] "RemoveContainer" containerID="0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac" Jul 24 01:02:24.955823 ip-10-0-134-162 kubenswrapper[2576]: E0724 01:02:24.955804 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac\": container with ID starting with 0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac not found: ID does not exist" containerID="0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac" Jul 24 01:02:24.955870 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.955832 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac"} err="failed to get container status \"0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac\": rpc error: code = NotFound desc = could not find container \"0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac\": container with ID starting with 0bf51ce463fa838e7fdab83ad2bc561fa3c45efcb780fd6a66953cba1ab1d2ac not found: ID does not exist" Jul 24 01:02:24.955870 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.955849 2576 scope.go:117] "RemoveContainer" containerID="66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1" Jul 24 01:02:24.956110 ip-10-0-134-162 kubenswrapper[2576]: E0724 01:02:24.956092 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1\": container with ID starting with 66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1 not found: ID does not exist" containerID="66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1" Jul 24 01:02:24.956159 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.956125 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1"} err="failed to get container status \"66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1\": rpc error: code = NotFound desc = could not find container \"66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1\": container with ID starting with 66f36190679828fb3946a233cf2a3a9ac684934ff32eaff08e284a8a139821f1 not found: ID does not exist" Jul 24 01:02:24.960414 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.960388 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl"] Jul 24 01:02:24.962538 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.962517 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:24.962538 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.962538 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:24.962669 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.962548 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:24.962669 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.962558 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:24.962669 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.962566 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxvvr\" (UniqueName: \"kubernetes.io/projected/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-kube-api-access-hxvvr\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:24.962669 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.962576 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:24.962669 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.962584 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c2b88b-9b1e-4bba-b7bb-0c285e292a34-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:24.963863 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:24.963842 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-86b8da-openshift-default-99f785c46-6fxvl"] Jul 24 01:02:26.002326 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:26.002290 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" path="/var/lib/kubelet/pods/55c2b88b-9b1e-4bba-b7bb-0c285e292a34/volumes" Jul 24 01:02:33.483856 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:33.483817 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 01:02:41.755861 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:41.755817 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9"] Jul 24 01:02:41.756560 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:41.756525 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" containerID="cri-o://2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0" gracePeriod=30 Jul 24 01:02:46.631756 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:46.631714 2576 patch_prober.go:28] interesting pod/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 01:02:46.632179 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:46.631801 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 01:02:47.017749 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.017722 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 01:02:47.058880 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.058848 2576 generic.go:356] "Generic (PLEG): container finished" podID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerID="2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0" exitCode=0 Jul 24 01:02:47.059069 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.058898 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" Jul 24 01:02:47.059069 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.058934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" event={"ID":"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1","Type":"ContainerDied","Data":"2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0"} Jul 24 01:02:47.059069 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.059007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9" event={"ID":"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1","Type":"ContainerDied","Data":"642a9491f4cb497977395fa5ce8534e732caf27868aa2ee0b761770858f8d368"} Jul 24 01:02:47.059069 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.059031 2576 scope.go:117] "RemoveContainer" containerID="2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0" Jul 24 01:02:47.069935 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.069917 2576 scope.go:117] "RemoveContainer" containerID="3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138" Jul 24 01:02:47.078520 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.078501 2576 scope.go:117] "RemoveContainer" containerID="2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0" Jul 24 01:02:47.078803 ip-10-0-134-162 kubenswrapper[2576]: E0724 01:02:47.078785 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0\": container with ID starting with 2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0 not found: ID does not exist" containerID="2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0" Jul 24 01:02:47.078854 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.078816 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0"} err="failed to get container status \"2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0\": rpc error: code = NotFound desc = could not find container \"2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0\": container with ID starting with 2f383bf7e6ccbba9b9a4f651e70160866c7d9429713d179e2b423189779e95e0 not found: ID does not exist" Jul 24 01:02:47.078854 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.078836 2576 scope.go:117] "RemoveContainer" containerID="3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138" Jul 24 01:02:47.079088 ip-10-0-134-162 kubenswrapper[2576]: E0724 01:02:47.079069 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138\": container with ID starting with 3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138 not found: ID does not exist" containerID="3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138" Jul 24 01:02:47.079155 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.079100 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138"} err="failed to get container status \"3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138\": rpc error: code = NotFound desc = could not find container \"3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138\": container with ID starting with 3e4d8db63790c93a1a57492262fdf9563e2ba7b2421f59bf01ba79d5404c6138 not found: ID does not exist" Jul 24 01:02:47.168870 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.168834 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-podinfo\") pod \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " Jul 24 01:02:47.169075 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.168893 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-credential-socket\") pod \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " Jul 24 01:02:47.169075 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.168910 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz8jz\" (UniqueName: \"kubernetes.io/projected/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-kube-api-access-kz8jz\") pod \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " Jul 24 01:02:47.169075 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.168944 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-token\") pod \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " Jul 24 01:02:47.169075 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.169001 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-data\") pod \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " Jul 24 01:02:47.169075 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.169037 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-envoy\") pod \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " Jul 24 01:02:47.169346 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.169095 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-workload-socket\") pod \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " Jul 24 01:02:47.169346 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.169137 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-workload-certs\") pod \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " Jul 24 01:02:47.169346 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.169168 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istiod-ca-cert\") pod \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\" (UID: \"7e5e619d-5ec8-4078-bee2-4cdf7addf4b1\") " Jul 24 01:02:47.169346 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.169209 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" (UID: "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:47.169545 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.169373 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" (UID: "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:47.169545 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.169401 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" (UID: "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:47.169650 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.169609 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:47.169650 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.169629 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:47.169650 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.169644 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:47.169805 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.169647 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" (UID: "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 01:02:47.170267 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.170235 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" (UID: "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:47.171343 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.171313 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" (UID: "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 01:02:47.171473 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.171453 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" (UID: "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 01:02:47.171535 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.171465 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" (UID: "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:02:47.171597 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.171531 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-kube-api-access-kz8jz" (OuterVolumeSpecName: "kube-api-access-kz8jz") pod "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" (UID: "7e5e619d-5ec8-4078-bee2-4cdf7addf4b1"). InnerVolumeSpecName "kube-api-access-kz8jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 01:02:47.270390 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.270294 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:47.270390 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.270327 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz8jz\" (UniqueName: \"kubernetes.io/projected/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-kube-api-access-kz8jz\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:47.270390 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.270337 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:47.270390 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.270345 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:47.270390 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.270353 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:47.270390 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.270360 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:02:47.381939 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.381903 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9"] Jul 24 01:02:47.385769 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:47.385742 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-38da25-openshift-default-6bd665b8b4-55dq9"] Jul 24 01:02:48.002205 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:48.002162 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" path="/var/lib/kubelet/pods/7e5e619d-5ec8-4078-bee2-4cdf7addf4b1/volumes" Jul 24 01:02:52.744871 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:52.744834 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck"] Jul 24 01:02:52.749908 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:52.749880 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-44edfc-29747580-rhfck"] Jul 24 01:02:54.002131 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:54.002097 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f75243-86a9-451f-95c2-534845c2c1d0" path="/var/lib/kubelet/pods/94f75243-86a9-451f-95c2-534845c2c1d0/volumes" Jul 24 01:02:55.690783 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:02:55.690742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-47jp2"] Jul 24 01:03:03.718993 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:03.718939 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp"] Jul 24 01:03:03.719493 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:03.719190 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" containerID="cri-o://bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84" gracePeriod=30 Jul 24 01:03:08.778759 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:08.778719 2576 patch_prober.go:28] interesting pod/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 24 01:03:08.779191 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:08.778777 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 24 01:03:08.980088 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:08.980059 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 01:03:09.064223 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.064184 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-workload-socket\") pod \"dcdfb9b3-1339-4b79-88de-5591c8520b57\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " Jul 24 01:03:09.064436 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.064240 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-data\") pod \"dcdfb9b3-1339-4b79-88de-5591c8520b57\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " Jul 24 01:03:09.064436 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.064261 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-envoy\") pod \"dcdfb9b3-1339-4b79-88de-5591c8520b57\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " Jul 24 01:03:09.064436 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.064286 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-workload-certs\") pod \"dcdfb9b3-1339-4b79-88de-5591c8520b57\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " Jul 24 01:03:09.064436 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.064351 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-token\") pod \"dcdfb9b3-1339-4b79-88de-5591c8520b57\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " Jul 24 01:03:09.064436 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.064379 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdfb9b3-1339-4b79-88de-5591c8520b57-istiod-ca-cert\") pod \"dcdfb9b3-1339-4b79-88de-5591c8520b57\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " Jul 24 01:03:09.064734 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.064463 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-credential-socket\") pod \"dcdfb9b3-1339-4b79-88de-5591c8520b57\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " Jul 24 01:03:09.064814 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.064795 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpnmq\" (UniqueName: \"kubernetes.io/projected/dcdfb9b3-1339-4b79-88de-5591c8520b57-kube-api-access-jpnmq\") pod \"dcdfb9b3-1339-4b79-88de-5591c8520b57\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " Jul 24 01:03:09.064940 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.064613 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dcdfb9b3-1339-4b79-88de-5591c8520b57" (UID: "dcdfb9b3-1339-4b79-88de-5591c8520b57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:03:09.064940 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.064729 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dcdfb9b3-1339-4b79-88de-5591c8520b57" (UID: "dcdfb9b3-1339-4b79-88de-5591c8520b57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:03:09.064940 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.064882 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-podinfo\") pod \"dcdfb9b3-1339-4b79-88de-5591c8520b57\" (UID: \"dcdfb9b3-1339-4b79-88de-5591c8520b57\") " Jul 24 01:03:09.065167 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.064983 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcdfb9b3-1339-4b79-88de-5591c8520b57-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dcdfb9b3-1339-4b79-88de-5591c8520b57" (UID: "dcdfb9b3-1339-4b79-88de-5591c8520b57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 01:03:09.065167 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.065055 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dcdfb9b3-1339-4b79-88de-5591c8520b57" (UID: "dcdfb9b3-1339-4b79-88de-5591c8520b57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:03:09.065167 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.065079 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-data" (OuterVolumeSpecName: "istio-data") pod "dcdfb9b3-1339-4b79-88de-5591c8520b57" (UID: "dcdfb9b3-1339-4b79-88de-5591c8520b57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:03:09.065493 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.065475 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-workload-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:03:09.065566 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.065499 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-data\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:03:09.065566 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.065513 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-workload-certs\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:03:09.065566 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.065525 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdfb9b3-1339-4b79-88de-5591c8520b57-istiod-ca-cert\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:03:09.065566 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.065539 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-credential-socket\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:03:09.066681 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.066654 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-token" (OuterVolumeSpecName: "istio-token") pod "dcdfb9b3-1339-4b79-88de-5591c8520b57" (UID: "dcdfb9b3-1339-4b79-88de-5591c8520b57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 01:03:09.066768 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.066697 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdfb9b3-1339-4b79-88de-5591c8520b57-kube-api-access-jpnmq" (OuterVolumeSpecName: "kube-api-access-jpnmq") pod "dcdfb9b3-1339-4b79-88de-5591c8520b57" (UID: "dcdfb9b3-1339-4b79-88de-5591c8520b57"). InnerVolumeSpecName "kube-api-access-jpnmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 01:03:09.066910 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.066888 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dcdfb9b3-1339-4b79-88de-5591c8520b57" (UID: "dcdfb9b3-1339-4b79-88de-5591c8520b57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 01:03:09.067056 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.067041 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dcdfb9b3-1339-4b79-88de-5591c8520b57" (UID: "dcdfb9b3-1339-4b79-88de-5591c8520b57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 01:03:09.166875 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.166830 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-token\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:03:09.166875 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.166868 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpnmq\" (UniqueName: \"kubernetes.io/projected/dcdfb9b3-1339-4b79-88de-5591c8520b57-kube-api-access-jpnmq\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:03:09.166875 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.166881 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-podinfo\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:03:09.167161 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.166890 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdfb9b3-1339-4b79-88de-5591c8520b57-istio-envoy\") on node \"ip-10-0-134-162.ec2.internal\" DevicePath \"\"" Jul 24 01:03:09.182007 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.181975 2576 generic.go:356] "Generic (PLEG): container finished" podID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerID="bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84" exitCode=0 Jul 24 01:03:09.182165 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.182034 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" event={"ID":"dcdfb9b3-1339-4b79-88de-5591c8520b57","Type":"ContainerDied","Data":"bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84"} Jul 24 01:03:09.182165 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.182083 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" event={"ID":"dcdfb9b3-1339-4b79-88de-5591c8520b57","Type":"ContainerDied","Data":"89096e0938a253341b5b233f939a53cf55d28030a4cca970932909e35f705d4f"} Jul 24 01:03:09.182165 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.182098 2576 scope.go:117] "RemoveContainer" containerID="bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84" Jul 24 01:03:09.182293 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.182085 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp" Jul 24 01:03:09.192078 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.192055 2576 scope.go:117] "RemoveContainer" containerID="938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f" Jul 24 01:03:09.204758 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.204730 2576 scope.go:117] "RemoveContainer" containerID="bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84" Jul 24 01:03:09.205151 ip-10-0-134-162 kubenswrapper[2576]: E0724 01:03:09.205127 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84\": container with ID starting with bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84 not found: ID does not exist" containerID="bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84" Jul 24 01:03:09.205244 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.205159 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84"} err="failed to get container status \"bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84\": rpc error: code = NotFound desc = could not find container \"bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84\": container with ID starting with bcdc3ed7e83de7e394cc504e735c9e1b6fdc0c675ff3e10baaf6e644f6f4dc84 not found: ID does not exist" Jul 24 01:03:09.205244 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.205181 2576 scope.go:117] "RemoveContainer" containerID="938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f" Jul 24 01:03:09.205457 ip-10-0-134-162 kubenswrapper[2576]: E0724 01:03:09.205437 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f\": container with ID starting with 938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f not found: ID does not exist" containerID="938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f" Jul 24 01:03:09.205500 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.205465 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f"} err="failed to get container status \"938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f\": rpc error: code = NotFound desc = could not find container \"938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f\": container with ID starting with 938dfbdc04c0da15244b0f6c146ba9aaf314edfa1ba184c9137905d2a1b4cd6f not found: ID does not exist" Jul 24 01:03:09.206906 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.206876 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp"] Jul 24 01:03:09.209381 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:09.209361 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-44edfc-openshift-default-6879748d76-qwfjp"] Jul 24 01:03:10.002120 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:10.002089 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" path="/var/lib/kubelet/pods/dcdfb9b3-1339-4b79-88de-5591c8520b57/volumes" Jul 24 01:03:10.769892 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:10.769863 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cff767bd6-8zhks_e127dce8-7b34-4e2e-a8d0-861ca1627220/copy-manifests/0.log" Jul 24 01:03:10.783642 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:10.783614 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cff767bd6-8zhks_e127dce8-7b34-4e2e-a8d0-861ca1627220/manager/0.log" Jul 24 01:03:11.408812 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:11.408783 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747550-d9v6f_6569a48f-baf4-422d-a7bd-d966036603fd/cleanup/0.log" Jul 24 01:03:11.519642 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:11.519613 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747565-4pp8v_ffd13eda-c959-4950-b88d-c70b141fb37f/cleanup/0.log" Jul 24 01:03:12.542398 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:12.542368 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls_c1c330cf-373a-4b2c-ba21-bd34dc15ce09/extract/0.log" Jul 24 01:03:12.549727 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:12.549690 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls_c1c330cf-373a-4b2c-ba21-bd34dc15ce09/util/0.log" Jul 24 01:03:12.556053 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:12.556034 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls_c1c330cf-373a-4b2c-ba21-bd34dc15ce09/pull/0.log" Jul 24 01:03:12.672014 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:12.671985 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn_b0a1f99d-14d2-4bea-8758-dfbaa40a1172/pull/0.log" Jul 24 01:03:12.678986 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:12.678962 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn_b0a1f99d-14d2-4bea-8758-dfbaa40a1172/extract/0.log" Jul 24 01:03:12.685883 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:12.685854 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn_b0a1f99d-14d2-4bea-8758-dfbaa40a1172/util/0.log" Jul 24 01:03:12.805676 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:12.805602 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk_af711c37-e5c5-4636-a72a-3dbb9d3f0176/util/0.log" Jul 24 01:03:12.812862 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:12.812838 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk_af711c37-e5c5-4636-a72a-3dbb9d3f0176/pull/0.log" Jul 24 01:03:12.819402 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:12.819381 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk_af711c37-e5c5-4636-a72a-3dbb9d3f0176/extract/0.log" Jul 24 01:03:12.935227 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:12.935198 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8_e8997ccf-869c-459d-ad1a-bbb82537e59e/pull/0.log" Jul 24 01:03:12.941431 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:12.941409 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8_e8997ccf-869c-459d-ad1a-bbb82537e59e/extract/0.log" Jul 24 01:03:12.947103 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:12.947085 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8_e8997ccf-869c-459d-ad1a-bbb82537e59e/util/0.log" Jul 24 01:03:13.317329 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:13.317300 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-t8htz_5cd36eda-9782-4c23-a860-f6a277925add/manager/0.log" Jul 24 01:03:13.555315 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:13.555274 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-vx7gf_2eb870b4-8978-43c6-bcb5-f4b96253989d/registry-server/0.log" Jul 24 01:03:13.789430 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:13.789407 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-47jp2_d10a100f-6ad4-4ea9-8a30-099f3dd405f8/limitador/0.log" Jul 24 01:03:13.904261 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:13.904231 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8rqgw_a9eb28d4-95d2-404d-ae10-2c4354213666/manager/0.log" Jul 24 01:03:14.269541 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:14.269456 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf_3cc4049d-653f-4735-bd2d-75581abf9202/istio-proxy/0.log" Jul 24 01:03:14.398070 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:14.398038 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-n29p5_5199901e-f6f0-4c4e-bd1e-dc1a1835766b/discovery/0.log" Jul 24 01:03:14.508551 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:14.508517 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c994f5cd9-9d7vp_fa64dff6-1958-4442-a3ab-71c1fbe98f86/kube-auth-proxy/0.log" Jul 24 01:03:14.752437 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:14.752407 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rfkdh_fa54590a-51b8-4e30-b65c-55089f26bf3c/istio-proxy/1.log" Jul 24 01:03:15.122723 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:15.122689 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-564867c99-6dnpx_7acb8462-421f-47e3-8bed-596f4f2d4d15/router/0.log" Jul 24 01:03:15.715875 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:15.715845 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-hb4wg_ed33a365-03ea-444d-89e3-38acf8eb2c2c/main/0.log" Jul 24 01:03:21.936815 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.936784 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sslln/must-gather-8kzcv"] Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937141 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bac8011-7aa9-407e-9249-47ade029c834" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937153 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bac8011-7aa9-407e-9249-47ade029c834" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937161 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937167 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937179 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937185 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937191 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bac8011-7aa9-407e-9249-47ade029c834" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937196 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bac8011-7aa9-407e-9249-47ade029c834" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937202 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937208 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937216 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937221 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937226 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937231 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937239 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937245 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" Jul 24 01:03:21.937246 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937254 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937259 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937310 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937318 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937325 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5e619d-5ec8-4078-bee2-4cdf7addf4b1" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937330 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bac8011-7aa9-407e-9249-47ade029c834" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937335 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bac8011-7aa9-407e-9249-47ade029c834" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937341 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937345 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937423 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937429 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937477 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2021f4-39cd-4772-8a57-f92d11e297f4" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937484 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c2b88b-9b1e-4bba-b7bb-0c285e292a34" containerName="istio-proxy" Jul 24 01:03:21.937788 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.937492 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcdfb9b3-1339-4b79-88de-5591c8520b57" containerName="istio-proxy" Jul 24 01:03:21.941785 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.941768 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sslln/must-gather-8kzcv" Jul 24 01:03:21.944436 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.944405 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sslln\"/\"openshift-service-ca.crt\"" Jul 24 01:03:21.944595 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.944451 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sslln\"/\"kube-root-ca.crt\"" Jul 24 01:03:21.944595 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.944468 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sslln\"/\"default-dockercfg-qwwxr\"" Jul 24 01:03:21.951623 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:21.951601 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sslln/must-gather-8kzcv"] Jul 24 01:03:22.090263 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:22.090227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgktx\" (UniqueName: \"kubernetes.io/projected/b381dfe4-6403-4cb8-8520-f7cff3484009-kube-api-access-fgktx\") pod \"must-gather-8kzcv\" (UID: \"b381dfe4-6403-4cb8-8520-f7cff3484009\") " pod="openshift-must-gather-sslln/must-gather-8kzcv" Jul 24 01:03:22.090441 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:22.090270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b381dfe4-6403-4cb8-8520-f7cff3484009-must-gather-output\") pod \"must-gather-8kzcv\" (UID: \"b381dfe4-6403-4cb8-8520-f7cff3484009\") " pod="openshift-must-gather-sslln/must-gather-8kzcv" Jul 24 01:03:22.191617 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:22.191524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgktx\" (UniqueName: \"kubernetes.io/projected/b381dfe4-6403-4cb8-8520-f7cff3484009-kube-api-access-fgktx\") pod \"must-gather-8kzcv\" (UID: \"b381dfe4-6403-4cb8-8520-f7cff3484009\") " pod="openshift-must-gather-sslln/must-gather-8kzcv" Jul 24 01:03:22.191617 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:22.191566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b381dfe4-6403-4cb8-8520-f7cff3484009-must-gather-output\") pod \"must-gather-8kzcv\" (UID: \"b381dfe4-6403-4cb8-8520-f7cff3484009\") " pod="openshift-must-gather-sslln/must-gather-8kzcv" Jul 24 01:03:22.191870 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:22.191853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b381dfe4-6403-4cb8-8520-f7cff3484009-must-gather-output\") pod \"must-gather-8kzcv\" (UID: \"b381dfe4-6403-4cb8-8520-f7cff3484009\") " pod="openshift-must-gather-sslln/must-gather-8kzcv" Jul 24 01:03:22.200023 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:22.199990 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgktx\" (UniqueName: \"kubernetes.io/projected/b381dfe4-6403-4cb8-8520-f7cff3484009-kube-api-access-fgktx\") pod \"must-gather-8kzcv\" (UID: \"b381dfe4-6403-4cb8-8520-f7cff3484009\") " pod="openshift-must-gather-sslln/must-gather-8kzcv" Jul 24 01:03:22.252513 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:22.252481 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sslln/must-gather-8kzcv" Jul 24 01:03:22.390017 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:22.389365 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sslln/must-gather-8kzcv"] Jul 24 01:03:22.393132 ip-10-0-134-162 kubenswrapper[2576]: W0724 01:03:22.393103 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb381dfe4_6403_4cb8_8520_f7cff3484009.slice/crio-0fcfe27134de2659219adaa8c14a7ac25e0f39f12c1442c87d6ed7f104d5b2ef WatchSource:0}: Error finding container 0fcfe27134de2659219adaa8c14a7ac25e0f39f12c1442c87d6ed7f104d5b2ef: Status 404 returned error can't find the container with id 0fcfe27134de2659219adaa8c14a7ac25e0f39f12c1442c87d6ed7f104d5b2ef Jul 24 01:03:22.395280 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:22.395256 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 01:03:23.262512 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:23.262474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sslln/must-gather-8kzcv" event={"ID":"b381dfe4-6403-4cb8-8520-f7cff3484009","Type":"ContainerStarted","Data":"0fcfe27134de2659219adaa8c14a7ac25e0f39f12c1442c87d6ed7f104d5b2ef"} Jul 24 01:03:26.282976 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:26.282850 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sslln/must-gather-8kzcv" event={"ID":"b381dfe4-6403-4cb8-8520-f7cff3484009","Type":"ContainerStarted","Data":"240ce5652f45063be28385ac0c77c5fa18e225fa7285c515e68b3b0a59050c7f"} Jul 24 01:03:26.282976 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:26.282899 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sslln/must-gather-8kzcv" event={"ID":"b381dfe4-6403-4cb8-8520-f7cff3484009","Type":"ContainerStarted","Data":"696cce751a78a739d77efcb9217bb1c4d00400e6f0868621a57fd359977d0ece"} Jul 24 01:03:26.306083 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:26.306014 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sslln/must-gather-8kzcv" podStartSLOduration=1.674661095 podStartE2EDuration="5.305993606s" totalImagesPullingTime="3.631332511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:03:26.304195024 +0000 UTC m=+2982.877291827" watchObservedRunningTime="2026-07-24 01:03:26.305993606 +0000 UTC m=+2982.879090415" Jul 24 01:03:27.749625 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:27.749585 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vncds_3c21f04e-dbb0-4743-831f-462ea9ffcd9c/global-pull-secret-syncer/0.log" Jul 24 01:03:27.842571 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:27.842538 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mw2d4_1c13136c-d5dd-438f-831e-367fcff88d3a/konnectivity-agent/0.log" Jul 24 01:03:27.928591 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:27.928561 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-162.ec2.internal_facb852d64cc5d31f3a76f06db558e9e/haproxy/0.log" Jul 24 01:03:32.461111 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:32.461076 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls_c1c330cf-373a-4b2c-ba21-bd34dc15ce09/extract/0.log" Jul 24 01:03:32.485213 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:32.485180 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls_c1c330cf-373a-4b2c-ba21-bd34dc15ce09/util/0.log" Jul 24 01:03:32.509366 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:32.509332 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759p68ls_c1c330cf-373a-4b2c-ba21-bd34dc15ce09/pull/0.log" Jul 24 01:03:32.547035 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:32.546980 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn_b0a1f99d-14d2-4bea-8758-dfbaa40a1172/extract/0.log" Jul 24 01:03:32.576477 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:32.576447 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn_b0a1f99d-14d2-4bea-8758-dfbaa40a1172/util/0.log" Jul 24 01:03:32.612504 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:32.612477 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk7dn_b0a1f99d-14d2-4bea-8758-dfbaa40a1172/pull/0.log" Jul 24 01:03:32.649073 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:32.649027 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk_af711c37-e5c5-4636-a72a-3dbb9d3f0176/extract/0.log" Jul 24 01:03:32.673401 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:32.673377 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk_af711c37-e5c5-4636-a72a-3dbb9d3f0176/util/0.log" Jul 24 01:03:32.701322 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:32.701294 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735plnk_af711c37-e5c5-4636-a72a-3dbb9d3f0176/pull/0.log" Jul 24 01:03:32.733868 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:32.733722 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8_e8997ccf-869c-459d-ad1a-bbb82537e59e/extract/0.log" Jul 24 01:03:32.766130 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:32.766079 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8_e8997ccf-869c-459d-ad1a-bbb82537e59e/util/0.log" Jul 24 01:03:32.796086 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:32.796057 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dgzs8_e8997ccf-869c-459d-ad1a-bbb82537e59e/pull/0.log" Jul 24 01:03:33.502078 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:33.502030 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-t8htz_5cd36eda-9782-4c23-a860-f6a277925add/manager/0.log" Jul 24 01:03:33.574609 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:33.574559 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-vx7gf_2eb870b4-8978-43c6-bcb5-f4b96253989d/registry-server/0.log" Jul 24 01:03:33.822828 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:33.822802 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-47jp2_d10a100f-6ad4-4ea9-8a30-099f3dd405f8/limitador/0.log" Jul 24 01:03:33.914880 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:33.914832 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8rqgw_a9eb28d4-95d2-404d-ae10-2c4354213666/manager/0.log" Jul 24 01:03:36.119539 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:36.119503 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkhtq_63a819a0-6c03-4aec-9c03-4fa662e3c100/node-exporter/0.log" Jul 24 01:03:36.150170 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:36.150073 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkhtq_63a819a0-6c03-4aec-9c03-4fa662e3c100/kube-rbac-proxy/0.log" Jul 24 01:03:36.172174 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:36.172141 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkhtq_63a819a0-6c03-4aec-9c03-4fa662e3c100/init-textfile/0.log" Jul 24 01:03:36.503897 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:36.503810 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7896f5b49c-h8wnl_3d15a62c-f665-4abb-ad18-e97811f54a3b/telemeter-client/0.log" Jul 24 01:03:36.526816 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:36.526784 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7896f5b49c-h8wnl_3d15a62c-f665-4abb-ad18-e97811f54a3b/reload/0.log" Jul 24 01:03:36.548281 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:36.548255 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7896f5b49c-h8wnl_3d15a62c-f665-4abb-ad18-e97811f54a3b/kube-rbac-proxy/0.log" Jul 24 01:03:37.841195 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:37.841155 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-stcch_dabcdd63-5694-4208-826b-6682b43c826d/networking-console-plugin/0.log" Jul 24 01:03:37.927547 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:37.927503 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l"] Jul 24 01:03:37.934452 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:37.934414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:37.941124 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:37.941093 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l"] Jul 24 01:03:37.961987 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:37.958353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd5qt\" (UniqueName: \"kubernetes.io/projected/d0a15431-1e28-4681-9c20-d27fc17d4bb8-kube-api-access-cd5qt\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:37.961987 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:37.958488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0a15431-1e28-4681-9c20-d27fc17d4bb8-proc\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:37.961987 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:37.958579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0a15431-1e28-4681-9c20-d27fc17d4bb8-sys\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:37.961987 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:37.958615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0a15431-1e28-4681-9c20-d27fc17d4bb8-podres\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:37.961987 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:37.958661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0a15431-1e28-4681-9c20-d27fc17d4bb8-lib-modules\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:38.059520 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.059479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd5qt\" (UniqueName: \"kubernetes.io/projected/d0a15431-1e28-4681-9c20-d27fc17d4bb8-kube-api-access-cd5qt\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:38.059847 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.059825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0a15431-1e28-4681-9c20-d27fc17d4bb8-proc\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:38.060121 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.060038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0a15431-1e28-4681-9c20-d27fc17d4bb8-sys\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:38.060520 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.060488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0a15431-1e28-4681-9c20-d27fc17d4bb8-podres\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:38.060626 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.060300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0a15431-1e28-4681-9c20-d27fc17d4bb8-sys\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:38.060626 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.060528 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0a15431-1e28-4681-9c20-d27fc17d4bb8-lib-modules\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:38.060626 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.060399 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0a15431-1e28-4681-9c20-d27fc17d4bb8-proc\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:38.060763 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.060635 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0a15431-1e28-4681-9c20-d27fc17d4bb8-lib-modules\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:38.060763 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.060649 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0a15431-1e28-4681-9c20-d27fc17d4bb8-podres\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:38.070464 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.070433 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd5qt\" (UniqueName: \"kubernetes.io/projected/d0a15431-1e28-4681-9c20-d27fc17d4bb8-kube-api-access-cd5qt\") pod \"perf-node-gather-daemonset-mm72l\" (UID: \"d0a15431-1e28-4681-9c20-d27fc17d4bb8\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:38.249328 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.249232 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:38.454195 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.454068 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l"] Jul 24 01:03:38.459485 ip-10-0-134-162 kubenswrapper[2576]: W0724 01:03:38.459429 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd0a15431_1e28_4681_9c20_d27fc17d4bb8.slice/crio-8e12fd307f97694e9edae2663e7437c4092000f7f1f46229007f4123ecae82c6 WatchSource:0}: Error finding container 8e12fd307f97694e9edae2663e7437c4092000f7f1f46229007f4123ecae82c6: Status 404 returned error can't find the container with id 8e12fd307f97694e9edae2663e7437c4092000f7f1f46229007f4123ecae82c6 Jul 24 01:03:38.517889 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.517859 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 01:03:38.522585 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:38.522561 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/2.log" Jul 24 01:03:39.084056 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:39.084027 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-7sbfn_a36a2229-e2b1-4935-81ff-b2ff0de97247/download-server/0.log" Jul 24 01:03:39.402866 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:39.402780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" event={"ID":"d0a15431-1e28-4681-9c20-d27fc17d4bb8","Type":"ContainerStarted","Data":"fea2f03976f1e93e7380e22a46b3908e8411c2136be991ab08e7fe69c3d5c533"} Jul 24 01:03:39.402866 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:39.402855 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:39.402866 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:39.402876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" event={"ID":"d0a15431-1e28-4681-9c20-d27fc17d4bb8","Type":"ContainerStarted","Data":"8e12fd307f97694e9edae2663e7437c4092000f7f1f46229007f4123ecae82c6"} Jul 24 01:03:39.419834 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:39.419771 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" podStartSLOduration=2.419755261 podStartE2EDuration="2.419755261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:03:39.418839253 +0000 UTC m=+2995.991936059" watchObservedRunningTime="2026-07-24 01:03:39.419755261 +0000 UTC m=+2995.992852063" Jul 24 01:03:39.731498 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:39.731414 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-vrzvd_91bc9290-4a98-4019-8a78-d35a28b84949/volume-data-source-validator/0.log" Jul 24 01:03:40.776471 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:40.776442 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nw9wd_03241e12-da01-4d42-abc0-30f73c4bb33b/dns/0.log" Jul 24 01:03:40.801816 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:40.801790 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nw9wd_03241e12-da01-4d42-abc0-30f73c4bb33b/kube-rbac-proxy/0.log" Jul 24 01:03:40.872358 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:40.872328 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8lgmt_301f4b24-c1c9-4754-ae99-e606ac261c2e/dns-node-resolver/0.log" Jul 24 01:03:41.486661 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:41.486627 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-cf7444644-zwzcq_459d4439-0f5a-4a4e-a30b-0918f09496d0/registry/0.log" Jul 24 01:03:41.509747 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:41.509717 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h6xgq_f1261605-62c9-473d-bf74-156b31afd9c6/node-ca/0.log" Jul 24 01:03:42.480224 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:42.480192 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf_3cc4049d-653f-4735-bd2d-75581abf9202/istio-proxy/0.log" Jul 24 01:03:42.670738 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:42.670707 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-n29p5_5199901e-f6f0-4c4e-bd1e-dc1a1835766b/discovery/0.log" Jul 24 01:03:42.693420 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:42.693384 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c994f5cd9-9d7vp_fa64dff6-1958-4442-a3ab-71c1fbe98f86/kube-auth-proxy/0.log" Jul 24 01:03:42.749716 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:42.749592 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rfkdh_fa54590a-51b8-4e30-b65c-55089f26bf3c/istio-proxy/0.log" Jul 24 01:03:42.785467 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:42.785439 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rfkdh_fa54590a-51b8-4e30-b65c-55089f26bf3c/istio-proxy/1.log" Jul 24 01:03:42.891829 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:42.891766 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-564867c99-6dnpx_7acb8462-421f-47e3-8bed-596f4f2d4d15/router/0.log" Jul 24 01:03:43.655345 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:43.655310 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4r25j_0a0428dc-cc75-43a6-8039-356a160b1b86/serve-healthcheck-canary/0.log" Jul 24 01:03:44.206450 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:44.206403 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-pnv5b_86ed4490-c0cd-45a0-bfe0-ba5730e4b621/insights-operator/0.log" Jul 24 01:03:44.206664 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:44.206636 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-pnv5b_86ed4490-c0cd-45a0-bfe0-ba5730e4b621/insights-operator/1.log" Jul 24 01:03:44.227252 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:44.227226 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 01:03:44.232243 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:44.232218 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-r6gpd_6015e731-af19-441b-8bd1-19f859ed77bb/console-operator/1.log" Jul 24 01:03:44.297297 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:44.297265 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k66cd_04e9e66a-b4ae-4fe2-a876-440487e20e2b/kube-rbac-proxy/0.log" Jul 24 01:03:44.318219 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:44.318186 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k66cd_04e9e66a-b4ae-4fe2-a876-440487e20e2b/exporter/0.log" Jul 24 01:03:44.338941 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:44.338913 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k66cd_04e9e66a-b4ae-4fe2-a876-440487e20e2b/extractor/0.log" Jul 24 01:03:45.422581 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:45.422553 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-mm72l" Jul 24 01:03:47.480534 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:47.480396 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cff767bd6-8zhks_e127dce8-7b34-4e2e-a8d0-861ca1627220/manager/0.log" Jul 24 01:03:47.499867 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:47.499824 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cff767bd6-8zhks_e127dce8-7b34-4e2e-a8d0-861ca1627220/copy-manifests/0.log" Jul 24 01:03:48.454211 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:48.454128 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf_3cc4049d-653f-4735-bd2d-75581abf9202/istio-proxy/0.log" Jul 24 01:03:48.620809 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:48.620770 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-n29p5_5199901e-f6f0-4c4e-bd1e-dc1a1835766b/discovery/0.log" Jul 24 01:03:48.630897 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:48.630871 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c994f5cd9-9d7vp_fa64dff6-1958-4442-a3ab-71c1fbe98f86/kube-auth-proxy/0.log" Jul 24 01:03:48.678000 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:48.677972 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rfkdh_fa54590a-51b8-4e30-b65c-55089f26bf3c/istio-proxy/0.log" Jul 24 01:03:48.713304 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:48.713227 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rfkdh_fa54590a-51b8-4e30-b65c-55089f26bf3c/istio-proxy/1.log" Jul 24 01:03:48.785787 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:48.785755 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-564867c99-6dnpx_7acb8462-421f-47e3-8bed-596f4f2d4d15/router/0.log" Jul 24 01:03:48.852583 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:48.852554 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-4vml8_a5718c65-142e-4729-b61d-50e9fca7d888/openshift-lws-operator/0.log" Jul 24 01:03:50.217233 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:50.217143 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-lpwsf_3cc4049d-653f-4735-bd2d-75581abf9202/istio-proxy/0.log" Jul 24 01:03:50.397311 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:50.397273 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-n29p5_5199901e-f6f0-4c4e-bd1e-dc1a1835766b/discovery/0.log" Jul 24 01:03:50.409153 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:50.409125 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c994f5cd9-9d7vp_fa64dff6-1958-4442-a3ab-71c1fbe98f86/kube-auth-proxy/0.log" Jul 24 01:03:50.456685 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:50.456650 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rfkdh_fa54590a-51b8-4e30-b65c-55089f26bf3c/istio-proxy/0.log" Jul 24 01:03:50.491969 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:50.491836 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rfkdh_fa54590a-51b8-4e30-b65c-55089f26bf3c/istio-proxy/1.log" Jul 24 01:03:50.568056 ip-10-0-134-162 kubenswrapper[2576]: I0724 01:03:50.568021 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-564867c99-6dnpx_7acb8462-421f-47e3-8bed-596f4f2d4d15/router/0.log"